Creating reftest-based unit tests

  • Revision slug: Creating_reftest-based_unit_tests
  • Revision title: Creating reftest-based unit tests
  • Revision id: 27078
  • Created:
  • Creator: Graememcc
  • Is current revision? No
  • Comment Rewrite section on privileges; 76 words added, 129 words removed

Revision Content

 

Your first reftest

The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.

This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a <blockquote> will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the "<blockquote>" element did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.

An example may make this clear. It is a silly example, but this will step you through creating your first reftest.

Step 1
For now you must check out and build the browser in order to run the tests. See the Build Documentation for details on doing that. Sorry about this, but the released builds and the nightly builds are built with the "--disable-tests" option and reftest will not work - see {{ Bug("369809") }}.
Step 2
Open a terminal window. Create a directory and make that your current directory (i.e. move to that directory).
Step 3
Create a file named foo.html with the following:
<html><head><title>reftest0001</title>
<body><strong>Hello!</strong></body>
</html>
Step 4
Create a file named bar.html with the following:
<html><head><title>reftest0001</title>
<body><b>Hello!</b></body>
</html>
Step 5
Create a file named reftest.list with the following:
== foo.html bar.html
You are now ready to run the test. Here is how I run this. Adapt this to your own platform.
% /bin/sh
$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2>&1 | grep REFTEST
REFTEST PASS: file:///Users/ray/moz/reftest0001.html
$

Congratulations! You have just created your first reftest!

In the above invocation of the browser, the "-P minefield1" made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See Firefox Help for more information on setting up profiles.) The re-direct and the grep reduce the amount of excess output from the browser. If you built a debug version of the browser, there can be a lot of extra console output. The reftest.list file can be named whatever you want, not necessarily reftest.list.

More to do

Create more reftests. New tests can be added to the reftest.list file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.

include ../other.list
== foo.html bar.html
!= aaa.html bbb.html

The first line, as one might expect, includes another manifest. The second line should look familiar. It says that foo.html and bar.html should produce visually identical output. The third line says that aaa.html and bbb.html should NOT produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.

There is one thing about automated tests that may not be obvious. There is really no way to construct a test that is too small. If you want to check something, and it seems trivial, that is ok. The cost of adding a new test to an automated suite is very, very low. For tests that are run manually, this is not true. The cost of thinking about and managing the execution of a manual test is fairly high. This is why manual tests tend to get longer, include more steps and ultimately become a long list that actually tests a lot of things.

So, create small tests. For example, it occurs to me that we assume that spaces between a element name and an attribute name have no effect, but do we know this is true? Who checks this? It is completely trivial, but so what. I can create 50 or 100 test files that have spaces between the element name and the attribute of a element for a bunch of different elements, add those to the list of tests to be run, and it causes no problems for anyone. Maybe it will actually take 500 test files to actually check this behavior. It really does not matter.

So, my point is, if you have an idea then create a test. Really. It would be better to have more tests than we need than too few.

Your second and third reftest

For these tests create the following files:

spaces1.html:

<html><head><title>spaces1</title></head>
<body>
X X
</body></html>

spaces2.html:

<html><head><title>spaces2</title></head>
<body>
X&nbsp;X
</body></html>

spaces3.html:

<html><head><title>spaces3</title></head>
<body>
X&nbsp;&nbsp;X
</body></html>

spaces4.html:

<html><head><title>spaces4</title></head>
<body>
X  X
</body></html>

reftests.txt:

== spaces1.html spaces2.html
!= spaces3.html spaces4.html

The first two files, spaces1.html and spaces2.html, are confirming only that a space (the character equal to 0x20 in ASCII) creates the same visual construct as the HTML entity for a non-breaking space. The second pair of files, spaces3.html and spaces4.html, confirms that two regular spaces do NOT produce the same visual construct as two non-breaking spaces.

When we run them, we see:

$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftests.txt 2>&1 | grep REFTEST
REFTEST PASS: file:///Users/ray/mo/spaces1.html
REFTEST PASS: (!=) file:///Users/ray/mo/spaces3.html
$ 

Fabulous!

Other comparisons

Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.

For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an <img> element.

More investigation into whether this will work is definitely warranted. Experiments with this have shown it to be not as easy as one would hope. We'll see.

Here is a list of reftest opportunities files in the source. These are files that have been checked in to be tested. Presumably, one was supposed to open the pages with a browser and look at them and see if they look right. It is hard to tell how many of these will be usable as reftests. If the file is associated with a bug, the bug should be examined. I have seen a case where the html file in the bug had a problem, but the checked-in version was "cleaned up" and not valid for testing.

Mozilla has used HTML generation tools in the past. The {{ Source("parser/htmlparser/tests/htmlgen/htmlgen.html", "htmlgen") }} tool is an example of this. Tools like this may be more useful now that we have reftest to exercise the files. It would also be useful to generate files that combine HTML and CSS in interesting or unusual ways.

Re-building reftest on Mac OS X

In order to make a change to reftest.js and see it take effect when you are building on a Mac OS X machine, one must do the following:

#!/bin/sh
cd layout/tools/reftest
d=../../../dist
rm -rf $d/Minefield.app/Contents/MacOS/chrome/reftest.*
rm -rf $d/bin/chrome/reftest.*
rm -rf $d/chrome-stage/chrome/reftest
make
cp $d/bin/chrome/reftest.jar $d/Minefield.app/Contents/MacOS/chrome
cp $d/bin/chrome/reftest.manifest $d/Minefield.app/Contents/MacOS/chrome

Reftests and elevated privileges

Reftests that you intend to check-in must not rely on behaviour that requires elevated privileges. The build process runs reftests in a profile that does not automatically grant elevated privileges: requesting them will cause the standard security alert to display, which will likely cause the machine running the test to hang or timeout.

Any tests that require such privileges to work correctly should be rewritten as Mochitests. The helper functions snapshotWindow and compareSnapshots are available in testing/mochitest/tests/SimpleTest/WindowSnapshot.js

References

{{ languages( { "ja": "ja/Creating_reftest-based_unit_tests" } ) }}

Revision Source

<p> </p>
<h3 name="Your_first_reftest">Your first reftest</h3>
<p>The reftest harness can be thought of as comparing two visual constructs. If the visual constructs produced by two files are identical, the test passes. If they differ, the test fails. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same visual construct.</p>
<p>This idea can seem odd when first encountered. Automated testing usually compares output against an invariant, a "gold standard", that is determined to be correct. If one has software that multiplies numbers, one wants a regression test to show that 2 * 2 continues to be calculated to be 4, not something similar to but not quite exactly 4. But an operating systems does change with time. It is not invariant. And a browser may change the visual effect produced by a tag while still being compliant with relevant standards. For example, the HTML 4.01 specification at the W3C specifies that text inside of a <code>&lt;blockquote&gt;</code> will be indented, but it does not specify the number of pixels of the indentation. If a browser changes the depth of the indenting and the visual construct is tested against an invariant, the test would appear to fail. But the test should not fail, unless the <code>"&lt;blockquote&gt;"</code> element did not cause any indentation at all. If a regression test harness has false failures, it makes the harness not trustworthy and the harness will not be used.</p>
<p>An example may make this clear. It is a silly example, but this will step you through creating your first reftest.</p>
<dl> <dt>Step 1</dt> <dd>For now you must check out and build the browser in order to run the tests. See the <a href="/En/Developer_Guide/Build_Instructions" title="En/Developer_Guide/Build_Instructions">Build Documentation</a> for details on doing that. Sorry about this, but the released builds and the nightly builds are built with the <code>"--disable-tests"</code> option and reftest will not work - see {{ Bug("369809") }}.</dd> <dt>Step 2</dt> <dd>Open a terminal window. Create a directory and make that your current directory (i.e. move to that directory).</dd> <dt>Step 3</dt> <dd>Create a file named <code>foo.html</code> with the following:</dd>
</dl>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;reftest0001&lt;/title&gt;
&lt;body&gt;&lt;strong&gt;Hello!&lt;/strong&gt;&lt;/body&gt;
&lt;/html&gt;
</pre>
<dl> <dt>Step 4</dt> <dd>Create a file named <code>bar.html</code> with the following:</dd>
</dl>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;reftest0001&lt;/title&gt;
&lt;body&gt;&lt;b&gt;Hello!&lt;/b&gt;&lt;/body&gt;
&lt;/html&gt;
</pre>
<dl> <dt>Step 5</dt> <dd>Create a file named <code>reftest.list</code> with the following:</dd>
</dl>
<pre class="eval">== foo.html bar.html
</pre>
<dl> <dd>You are now ready to run the test. Here is how I run this. Adapt this to your own platform.</dd>
</dl>
<pre class="eval">% /bin/sh
$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftest.list 2&gt;&amp;1 | grep REFTEST
REFTEST PASS: <a class=" external" href="file:///Users/ray/moz/reftest0001.html" rel="freelink">file:///Users/ray/moz/reftest0001.html</a>
$
</pre>
<p>Congratulations! You have just created your first reftest!</p>
<p>In the above invocation of the browser, the <code>"-P minefield1"</code> made sure that the test used a profile that had been set up for testing. This profile should not have personal or important data. (See <a class="external" href="http://www.mozilla.org/support/firefox/profile">Firefox Help</a> for more information on setting up profiles.) The re-direct and the <code>grep</code> reduce the amount of excess output from the browser. If you built a debug version of the browser, there can be a lot of extra console output. The <code>reftest.list</code> file can be named whatever you want, not necessarily <code>reftest.list</code>.</p>
<h3 name="More_to_do">More to do</h3>
<p>Create more reftests. New tests can be added to the <code>reftest.list</code> file, which can contain any number of tests. The file can include other things, but it does not get very complicated. Here is an example.</p>
<pre class="eval">include ../other.list
== foo.html bar.html
!= aaa.html bbb.html
</pre>
<p>The first line, as one might expect, includes another manifest. The second line should look familiar. It says that <code>foo.html</code> and <code>bar.html</code> should produce visually identical output. The third line says that <code>aaa.html</code> and <code>bbb.html</code> should <strong>NOT</strong> produce visually identical output. More information on this file can be found in the README.txt referenced below, which was written by the creator of the reftest tool.</p>
<p>There is one thing about automated tests that may not be obvious. There is really no way to construct a test that is too small. If you want to check something, and it seems trivial, that is ok. The cost of adding a new test to an automated suite is very, very low. For tests that are run manually, this is not true. The cost of thinking about and managing the execution of a manual test is fairly high. This is why manual tests tend to get longer, include more steps and ultimately become a long list that actually tests a lot of things.</p>
<p>So, create small tests. For example, it occurs to me that we assume that spaces between a element name and an attribute name have no effect, but do we know this is true? Who checks this? It is completely trivial, but so what. I can create 50 or 100 test files that have spaces between the element name and the attribute of a element for a bunch of different elements, add those to the list of tests to be run, and it causes no problems for anyone. Maybe it will actually take 500 test files to actually check this behavior. It really does not matter.</p>
<p>So, my point is, if you have an idea then create a test. Really. It would be better to have more tests than we need than too few.</p>
<h3 name="Your_second_and_third_reftest">Your second and third reftest</h3>
<p>For these tests create the following files:</p>
<p>spaces1.html:</p>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;spaces1&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
X X
&lt;/body&gt;&lt;/html&gt;
</pre>
<p>spaces2.html:</p>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;spaces2&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
X&amp;nbsp;X
&lt;/body&gt;&lt;/html&gt;
</pre>
<p>spaces3.html:</p>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;spaces3&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
X&amp;nbsp;&amp;nbsp;X
&lt;/body&gt;&lt;/html&gt;
</pre>
<p>spaces4.html:</p>
<pre class="eval">&lt;html&gt;&lt;head&gt;&lt;title&gt;spaces4&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
X  X
&lt;/body&gt;&lt;/html&gt;
</pre>
<p>reftests.txt:</p>
<pre class="eval">== spaces1.html spaces2.html
!= spaces3.html spaces4.html
</pre>
<p>The first two files, <code>spaces1.html</code> and <code>spaces2.html</code>, are confirming only that a space (the character equal to 0x20 in ASCII) creates the same visual construct as the HTML entity for a non-breaking space. The second pair of files, <code>spaces3.html</code> and <code>spaces4.html</code>, confirms that two regular spaces do <strong>NOT</strong> produce the same visual construct as two non-breaking spaces.</p>
<p>When we run them, we see:</p>
<pre class="eval">$ /Users/ray/mo/browser/mozilla/dist/MinefieldDebug.app/Contents/MacOS/firefox -P minefield1 -reftest ./reftests.txt 2&gt;&amp;1 | grep REFTEST
REFTEST PASS: <a class=" external" href="file:///Users/ray/mo/spaces1.html" rel="freelink">file:///Users/ray/mo/spaces1.html</a>
REFTEST PASS: (!=) <a class=" external" href="file:///Users/ray/mo/spaces3.html" rel="freelink">file:///Users/ray/mo/spaces3.html</a>
$ 
</pre>
<p>Fabulous!</p>
<h3 name="Other_comparisons">Other comparisons</h3>
<p>Note that it should also be possible to create a reftest that tests markup against an actual graphic image of the visual construct that should result. This is probably going to be a more fragile test, for the reasons described above. But it may be necessary.</p>
<p>For example, say that certain markup should produce a certain Sanskrit glyph. How does one test that? One should be able to "snap a picture" of the glyph as it should be displayed and then the reference page would include that graphic in an <code>&lt;img&gt;</code> element.</p>
<p>More investigation into whether this will work is definitely warranted. Experiments with this have shown it to be not as easy as one would hope. We'll see.</p>
<p>Here is a list of <a href="/en/reftest_opportunities_files" title="en/reftest_opportunities_files">reftest opportunities files</a> in the source. These are files that have been checked in to be tested. Presumably, one was supposed to open the pages with a browser and look at them and see if they look right. It is hard to tell how many of these will be usable as reftests. If the file is associated with a bug, the bug should be examined. I have seen a case where the html file in the bug had a problem, but the checked-in version was "cleaned up" and not valid for testing.</p>
<p>Mozilla has used HTML generation tools in the past. The {{ Source("parser/htmlparser/tests/htmlgen/htmlgen.html", "htmlgen") }} tool is an example of this. Tools like this may be more useful now that we have reftest to exercise the files. It would also be useful to generate files that combine HTML and CSS in interesting or unusual ways.</p>
<h3 name="Re-building_reftest_on_Mac_OS_X">Re-building reftest on Mac OS X</h3>
<p>In order to make a change to reftest.js and see it take effect when you are building on a Mac OS X machine, one must do the following:</p>
<pre class="eval">#!/bin/sh
cd layout/tools/reftest
d=../../../dist
rm -rf $d/Minefield.app/Contents/MacOS/chrome/reftest.*
rm -rf $d/bin/chrome/reftest.*
rm -rf $d/chrome-stage/chrome/reftest
make
cp $d/bin/chrome/reftest.jar $d/Minefield.app/Contents/MacOS/chrome
cp $d/bin/chrome/reftest.manifest $d/Minefield.app/Contents/MacOS/chrome
</pre>
<h3 name="Creating_reftests_for_check-in">Reftests and elevated privileges</h3>
<p>Reftests that you intend to check-in must not rely on behaviour that requires elevated privileges. The build process runs reftests in a profile that does not automatically grant elevated privileges: requesting them will cause the standard security alert to display, which will likely cause the machine running the test to hang or timeout.</p>
<p>Any tests that require such privileges to work correctly should be rewritten as <a href="/en/Mochitest" title="en/Mochitest">Mochitests</a>. The helper functions <code>snapshotWindow</code> and <code>compareSnapshots</code> are available in <a class=" external" href="http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/WindowSnapshot.js" title="http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/WindowSnapshot.js">testing/mochitest/tests/SimpleTest/WindowSnapshot.js</a></p>
<h3 name="References">References</h3>
<ul> <li><a class=" external" href="http://weblogs.mozillazine.org/qa/archives/2006/12/test_tool_spotlight_reftest.html" rel="freelink">http://weblogs.mozillazine.org/qa/ar...t_reftest.html</a></li> <li><a class=" external" href="http://www.nabble.com/Please-add-%22in-testsuite-%22-to-layout-bugs-when-resolving-them!-t2791203.html" rel="freelink">http://www.nabble.com/Please-add-%22...-t2791203.html</a></li> <li>{{ Source("layout/tools/reftest/README.txt") }}</li>
</ul>

<p>{{ languages( { "ja": "ja/Creating_reftest-based_unit_tests" } ) }}</p>
Revert to this revision