Your First Reftest
The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.
This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a "<blockquote>" will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the "<blockquote>" tag did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.
An example may make this clear. It is a silly example, but this will step you through creating your first reftest.
Step 1: One must have a copy of the browser that one has built. (See http://developer.mozilla.org/en/docs/Mozilla_Build_FAQ.) Sorry about this, but the released builds and the nightly builds are built with the "--disable-tests" option and reftest will not work. (See https://bugzilla.mozilla.org/show_bug.cgi?id=369809.)
Step 2: Open a terminal window. Create a directory and make that your current directory.
Step 3: Create a file named
foo.html with the following:
Step 4: Create a file named
bar.html with the following:
Step 5: Create a file named
reftest.list with the following:
== foo.html bar.html
One is now ready to run the test. Here is how I run this. Adapt this to your own platform.
% /bin/sh $ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2>&1 | grep REFTEST REFTEST PASS: file:///Users/ray/moz/reftest0001.html $
Congratulations! You have just created your first retest!
In the above invocation of the browser, the "-P minefield1" made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See http://www.mozilla.org/support/firefox/profile for more information on setting up profiles.) The re-direct and the
grep reduce the amount of excess output from the browser. If one builds the browser with debug enabled, there can be a lot of extra console output. The
reftest.list file does not follow a naming convention. It can be named whatever you want.
More to Do
Create more reftests. New tests can be added to the
reftest.list file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.
include ../other.list == foo.html bar.html != aaa.html bbb.html
The first line, as one might expect, includes another list. The second line should look familiar. It says that
bar.html should produce visually identical output. The third line says that
bbb.html should NOT produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.
Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.
For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an <img> tag.
More investigation into whether this will work is definitely warranted.