mozilla

Revision 27063 of Creating reftest-based unit tests

  • Revision slug: Creating_reftest-based_unit_tests
  • Revision title: Creating reftest-based unit tests
  • Revision id: 27063
  • Created:
  • Creator: Kray2
  • Is current revision? No
  • Comment first version of the first couple sections of this document

Revision Content

Your First Reftest

The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.

This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a "<blockquote>" will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the "<blockquote>" tag did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.

An example may make this clear. It is a silly example, but this will step you through creating your first reftest.

Step 1: One must have a copy of the browser that one has built. (See http://developer.mozilla.org/en/docs/Mozilla_Build_FAQ.) Sorry about this, but the released builds and the nightly builds are built with the "--disable-tests" option and reftest will not work. (See https://bugzilla.mozilla.org/show_bug.cgi?id=369809.)

Step 2: Open a terminal window. Create a directory and make that your current directory.

Step 3: Create a file named foo.html with the following:

<html><head><title>reftest0001</title><body><strong>Hello!</strong></body></html>

Step 4: Create a file named bar.html with the following:

<html><head><title>reftest0001</title><body><b>Hello!</b></body></html>

Step 5: Create a file named reftest.list with the following:

== foo.html bar.html

One is now ready to run the test. Here is how I run this. Adapt this to your own platform.

% /bin/sh
$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2>&1 | grep REFTEST
REFTEST PASS: file:///Users/ray/moz/reftest0001.html
$

Congratulations! You have just created your first retest!

In the above invocation of the browser, the "-P minefield1" made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See http://www.mozilla.org/support/firefox/profile for more information on setting up profiles.) The re-direct and the grep reduce the amount of excess output from the browser. If one builds the browser with debug enabled, there can be a lot of extra console output. The reftest.list file does not follow a naming convention. It can be named whatever you want.

More to Do

Create more reftests. New tests can be added to the reftest.list file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.

include ../other.list
== foo.html bar.html
!= aaa.html bbb.html

The first line, as one might expect, includes another list. The second line should look familiar. It says that foo.html and bar.html should produce visually identical output. The third line says that aaa.html and bbb.html should NOT produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.

Other Comparisons

Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.

For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an <img> tag.

More investigation into whether this will work is definitely warranted.

References

  • http://weblogs.mozillazine.org/qa/archives/2006/12/test_tool_spotlight_reftest.html
  • http://www.nabble.com/Please-add-%22in-testsuite-%22-to-layout-bugs-when-resolving-them!-t2791203.html
  • http://lxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt

Revision Source

<h3 name="Your_First_Reftest"> Your First Reftest </h3>
<p>The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.
</p><p>This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a "&lt;blockquote&gt;" will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the "&lt;blockquote&gt;" tag did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.
</p><p>An example may make this clear. It is a silly example, but this will step you through creating your first reftest.
</p><p>Step 1: One must have a copy of the browser that one has built. (See http://developer.mozilla.org/en/docs/Mozilla_Build_FAQ.) Sorry about this, but the released builds and the nightly builds are built with the "--disable-tests" option and reftest will not work. (See https://bugzilla.mozilla.org/show_bug.cgi?id=369809.)
</p><p>Step 2: Open a terminal window. Create a directory and make that your current directory.
</p><p>Step 3: Create a file named <code>foo.html</code> with the following:
</p>
<blockquote><pre>&lt;html&gt;&lt;head&gt;&lt;title&gt;reftest0001&lt;/title&gt;&lt;body&gt;&lt;strong&gt;Hello!&lt;/strong&gt;&lt;/body&gt;&lt;/html&gt;
</pre></blockquote>
<p>Step 4: Create a file named <code>bar.html</code> with the following:
</p>
<blockquote><pre>&lt;html&gt;&lt;head&gt;&lt;title&gt;reftest0001&lt;/title&gt;&lt;body&gt;&lt;b&gt;Hello!&lt;/b&gt;&lt;/body&gt;&lt;/html&gt;
</pre></blockquote>
<p>Step 5: Create a file named <code>reftest.list</code> with the following:
</p>
<blockquote><pre>== foo.html bar.html
</pre></blockquote>
<p>One is now ready to run the test. Here is how I run this. Adapt this to your own platform.
</p>
<blockquote><pre>% /bin/sh
$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2&gt;&amp;1 | grep REFTEST
REFTEST PASS: file:///Users/ray/moz/reftest0001.html
$
</pre></blockquote>
<p>Congratulations! You have just created your first retest!
</p><p>In the above invocation of the browser, the "-P minefield1" made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See http://www.mozilla.org/support/firefox/profile for more information on setting up profiles.) The re-direct and the <code>grep</code> reduce the amount of excess output from the browser. If one builds the browser with debug enabled, there can be a lot of extra console output. The <code>reftest.list</code> file does not follow a naming convention. It can be named whatever you want.
</p>
<h3 name="More_to_Do"> More to Do </h3>
<p>Create more reftests. New tests can be added to the <code>reftest.list</code> file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.
</p>
<blockquote><pre>include ../other.list
== foo.html bar.html
!= aaa.html bbb.html
</pre></blockquote>
<p>The first line, as one might expect, includes another list. The second line should look familiar. It says that <code>foo.html</code> and <code>bar.html</code> should produce visually identical output. The third line says that <code>aaa.html</code> and <code>bbb.html</code> should <strong>NOT</strong> produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.
</p>
<h3 name="Other_Comparisons"> Other Comparisons </h3>
<p>Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.
</p><p>For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an &lt;img&gt; tag.
</p><p>More investigation into whether this will work is definitely warranted.
</p>
<h3 name="References"> References </h3>
<ul><li> http://weblogs.mozillazine.org/qa/archives/2006/12/test_tool_spotlight_reftest.html
</li></ul>
<ul><li> http://www.nabble.com/Please-add-%22in-testsuite-%22-to-layout-bugs-when-resolving-them!-t2791203.html
</li></ul>
<ul><li> http://lxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt
</li></ul>
Revert to this revision