Your First Reftest
The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.
This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a "<blockquote>" will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the "<blockquote>" tag did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.
An example may make this clear. It is a silly example, but this will step you through creating your first reftest.
Step 1: One must have a copy of the browser that one has built. (See http://developer.mozilla.org/en/docs/Mozilla_Build_FAQ.) Sorry about this, but the released builds and the nightly builds are built with the "--disable-tests" option and reftest will not work. (See https://bugzilla.mozilla.org/show_bug.cgi?id=369809.)
Step 2: Open a terminal window. Create a directory and make that your current directory.
Step 3: Create a file named
foo.html with the following:
Step 4: Create a file named
bar.html with the following:
Step 5: Create a file named
reftest.list with the following:
== foo.html bar.html
One is now ready to run the test. Here is how I run this. Adapt this to your own platform.
% /bin/sh $ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2>&1 | grep REFTEST REFTEST PASS: file:///Users/ray/moz/reftest0001.html $
Congratulations! You have just created your first retest!
In the above invocation of the browser, the "-P minefield1" made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See http://www.mozilla.org/support/firefox/profile for more information on setting up profiles.) The re-direct and the
grep reduce the amount of excess output from the browser. If one builds the browser with debug enabled, there can be a lot of extra console output. The
reftest.list file does not follow a naming convention. It can be named whatever you want.
More to Do
Create more reftests. New tests can be added to the
reftest.list file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.
include ../other.list == foo.html bar.html != aaa.html bbb.html
The first line, as one might expect, includes another list. The second line should look familiar. It says that
bar.html should produce visually identical output. The third line says that
bbb.html should NOT produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.
There is one thing about automated tests that may not be obvious. There is really no way to construct a test that is too small. If you want to check something, and it seems trivial, that is ok. The cost of adding a new test to an automated suite is very, very low. For tests that are run manually, this is not true. The cost of thinking about and managing the execution of a manual test is fairly high. This is why manual tests tend to get longer, include more steps and ultimately become a long list that actually tests a lot of things.
So, create small tests. For example, it occurs to me that we assume that spaces between a tag name and an attribute name have no effect, but do we know this is true? Who checks this? It is completely trivial, but so what. I can create 50 or 100 test files that have spaces between the tag and the attribute of a tag for a bunch of different tags, add those to the list of tests to be run, and it causes no problems for anyone. Maybe it will actually take 500 test files to actually check this behavior. It really does not matter.
So, my point is, if you have an idea then create a test. Really. It would be better to have more tests than we need than too few.
Your Second and Third Reftest
For these tests create the following files:
<html><head><title>spaces1</title></head> <body> X X </body></html>
<html><head><title>spaces2</title></head> <body> X X </body></html>
<html><head><title>spaces3</title></head> <body> X X </body></html>
<html><head><title>spaces4</title></head> <body> X X </body></html>
== spaces1.html spaces2.html != spaces3.html spaces4.html
The first two files,
spaces2.html, are confirming only that a space (the character equal to 0x20 in ASCII) creates the same visual construct as the HTML entity for a non-breaking space. The second pair of files,
spaces4.html, confirms that two regular spaces do NOT produce the same visual construct as two non-breaking spaces.
When we run them, we see:
$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftests.txt 2>&1 | grep REFTEST REFTEST PASS: file:///Users/ray/mo/spaces1.html REFTEST PASS: (!=) file:///Users/ray/mo/spaces3.html $
Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.
For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an <img> tag.
More investigation into whether this will work is definitely warranted. Experiments with this have shown it to be not as easy as one would hope. We'll see.