Mozilla automated testing

  • Revision slug: Mozilla_automated_testing
  • Revision title: Mozilla automated testing
  • Revision id: 4168
  • Created:
  • Creator: Jesse
  • Is current revision? No
  • Comment 224 words added, 867 words removed

Revision Content

Note: It has been proposed that this page and Developing tests be merged.

This page gives an overview of options for automated testing available to Mozilla developers.

After building Firefox with --enable-tests, you can run any regression-test suite from your objdir with a simple make command.

Mozilla uses several test frameworks. They are listed here in rough order from the lowest-level unit tests to the highest-level system tests.

TBPL
code
Command Description
(B) make check Compiled-code tests can test non-scriptable interfaces, but are hard to write and maintain.
X make xpcshell-tests JavaScript code runs in xpcshell. Tests can access XPCOM components, but can't easily work with documents or windows.

 

J make jstestbrowser JavaScript engine regression tests.
C make crashtest A web page is loaded.
R make reftest A reftest consists of a pair of web pages. The test passes if the pages are painted identically. Ideal for testing visual web features.
M make mochitest-plain Mochitest pages are loaded in the browser with low privileges. Ideal for thoroughly testing a web feature.
Moth make mochitest-chrome Chrome mochitest pages are loaded with high privileges.
make mochitest-browser-chrome Browser chrome tests run in a browser window's scope. Ideal for testing front-end code directly.

With the exception of R and C, all individual tests define their own pass and fail conditions.

All test suites treat crashes, hangs, and (in debug builds) trace-refcnt leaks as failures. NS_ASSERTIONs are treated as fatal in X, annotatable failures in R/C/J, and ignored in M.

Most suites share a common test log format, so you can grep for UNEXPECTED to find failures in your run.

You can debug with EXTRA_TEST_ARGS='--debugger=gdb' make [suite].

Mozmill tests

Mozmill is a record/playback style automation tool for automating end user interaction with an application.  It can run on any application developed from Gecko 1.9.0 and later.  Tests are written in simple JavaScript against an automation API.

Further Reading

Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.

The log output format of the tests is documented at Test log format.

There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and Automated testing tips and tricks if you're looking for something to read.

These are some additional efforts underway:

  • You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM.
    • {{ Bug("343673") }} tracks one person's effort, but there has been no recent activity
    • {{ Bug("346703") }} contains one example of how this might be done
  • JSUnit can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges.
    • JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.
    • {{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.
    • Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.
    • Some XForms tests uses JsUnit, as an example

Existing Test Harnesses and Frameworks

(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})

{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}

Revision Source

<div class="note"><strong>Note:</strong> It has been proposed that this page and <a href="/en/Developing_Tests" title="en/Developing tests">Developing tests</a> be merged.</div>
<p>This page gives an overview of options for automated testing available to Mozilla developers.</p>
<p>After <a href="/En/Simple_Firefox_build">building Firefox</a> with --enable-tests, you can run any regression-test suite from your objdir with a simple make command.</p>
<p>Mozilla uses several test frameworks. They are listed here in rough order from the lowest-level unit tests to the highest-level system tests.</p>
<table border="1"> <thead> <tr> <th>TBPL<br> code</th> <th>Command</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>(B)</td> <td><code>make check</code></td> <td><a href="/en/Compiled-code_automated_tests">Compiled-code tests</a> can test non-scriptable interfaces, but are hard to write and maintain.</td> </tr> <tr> <td>X</td> <td><code>make xpcshell-tests</code></td> <td>JavaScript code runs in <a href="/en/XPConnect/xpcshell">xpcshell</a>. Tests can access XPCOM components, but can't easily work with documents or windows. <p> </p> </td> </tr> <tr> <td>J</td> <td><code>make jstestbrowser</code></td> <td>JavaScript engine regression tests.</td> </tr> <tr> <td>C</td> <td><code>make crashtest</code></td> <td>A web page is loaded.</td> </tr> <tr> <td>R</td> <td><code>make reftest</code></td> <td>A <a href="/en/Creating_reftest-based_unit_tests">reftest</a> consists of a <em>pair</em> of web pages. The test passes if the pages are painted identically. Ideal for testing visual web features.</td> </tr> <tr> <td>M</td> <td><code>make mochitest-plain</code></td> <td><a href="/en/Mochitest">Mochitest pages</a> are loaded in the browser with low privileges. Ideal for thoroughly testing a web feature.</td> </tr> <tr> <td rowspan="2">Moth</td> <td><code>make mochitest-chrome</code></td> <td><a href="/en/Chrome_tests">Chrome mochitest pages</a> are loaded with high privileges.</td> </tr> <tr> <td><code>make mochitest-browser-chrome</code></td> <td><a href="/en/Browser_chrome_tests">Browser chrome tests</a> run in a browser window's scope. Ideal for testing front-end code directly.</td> </tr> </tbody>
</table>
<p>With the exception of R and C, all individual tests define their own pass and fail conditions.</p>
<p>All test suites treat crashes, hangs, and (in debug builds) <a class=" link-https" href="https://wiki.mozilla.org/Performance:Leak_Tools">trace-refcnt leaks</a> as failures. NS_ASSERTIONs are treated as fatal in X, annotatable failures in R/C/J, and ignored in M.</p>
<p>Most suites share a common <a href="/en/Test_log_format" rel="internal">test log format</a>, so you can grep for UNEXPECTED to find failures in your run.</p>
<p>You can debug with <code>EXTRA_TEST_ARGS='--debugger=gdb' make [suite]</code>.</p>
<h4 name="Crash_tests">Mozmill tests</h4>
<p><a class="internal" href="/en/Mozmill_Tests" title="en/Mozmill Tests">Mozmill</a> is a record/playback style automation tool for automating end user interaction with an application.  It can run on any application developed from Gecko 1.9.0 and later.  Tests are written in simple JavaScript against an automation API.</p>
<h3 name="Further_Reading">Further Reading</h3>
<p>Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.</p>
<p>The log output format of the tests is documented at <a href="/en/Test_log_format" title="https://developer.mozilla.org/en/Test_log_format">Test log format</a>.</p>
<p>There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and <a href="/en/Automated_testing_tips_and_tricks" title="en/Automated_testing_tips_and_tricks">Automated testing tips and tricks</a> if you're looking for something to read.</p>
<p>These are some additional efforts underway:</p>
<ul> <li>You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM. <ul> <li><em>{{ Bug("343673") }} tracks one person's effort, but there has been no recent activity</em></li> <li><em>{{ Bug("346703") }} contains one example of how this might be done</em></li> </ul> </li> <li><a class="external" href="http://www.jsunit.net/">JSUnit</a> can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges. <ul> <li>JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.</li> <li>{{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.</li> <li>Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.</li> <li>Some <a class="external" href="http://beaufour.dk/xftst/">XForms tests</a> uses JsUnit, as an example</li> </ul> </li>
</ul>
<h3 name="Existing_Test_Harnesses_and_Frameworks">Existing Test Harnesses and Frameworks</h3>
<p>(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})</p>
<ul> <li><a class="external" href="http://wiki.mozilla.org/Performance:Tinderbox_Tests">Tinderbox performance tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/browser/components/places/tests/">Places test script</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/netwerk/test/unit/">Netwerk unit tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/js/tests/">javascript tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/security/nss/tests/">nss tests</a></li> <li><a class="external" href="http://landfill.mozilla.org/mxr-test/mozilla/source/nsprpub/pr/tests/">nspr tests</a></li> <li><a class="external" href="http://www.mozilla.org/newlayout/doc/regression_tests.html">layout tests - diff of frame tree dump against golden master</a></li> <li><a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=301260">bz's copy of netwerk tests for xmlserializer</a></li> <li><a class="external" href="http://www.w3.org/DOM/Test/">W3C DOM Tests</a> uses <a class="external" href="http://www.edwardh.com/jsunit/">jsunit</a></li> <li><a class="external" href="http://wiki.mozilla.org/XSLT_Tests">XSLT Tests</a></li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST" rel="freelink">http://hixie.ch/tests/MANIFEST</a> all hixie's tests</li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST-visual" rel="freelink">http://hixie.ch/tests/MANIFEST-visual</a> subset of hixie's tests that is non-interactive</li> <li><a class="external" href="http://www.allpeers.com/blog/2005/09/28/foxunit-unit-test-framework-for-firefox/">FoxUnit</a> - jUnit-like harness for Firefox, by the AllPeers folks</li>
</ul>
<p>{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}</p>
Revert to this revision