Mozilla automated testing

  • Revision slug: Mozilla_automated_testing
  • Revision title: Mozilla automated testing
  • Revision id: 4171
  • Created:
  • Creator: Jesse
  • Is current revision? No
  • Comment 27 words added, 44 words removed

Revision Content

Note: It has been proposed that this page and Developing tests be merged.

This page gives an overview of options for automated testing available to Mozilla developers.

After building Firefox with --enable-tests, you can run any regression-test suite from your objdir with a simple make command.

Mozilla uses several test frameworks. They are listed here in rough order from the lowest-level unit tests to the highest-level system tests.

TBPL
code
Command Description
(B) make check Compiled-code tests can test non-scriptable interfaces, but are hard to write and maintain.
X make xpcshell-tests JavaScript code runs in xpcshell. Tests can access XPCOM components, but can't easily work with documents or windows.
J make jstestbrowser JavaScript engine regression tests.
C make crashtest A web page is loaded.
R make reftest A reftest consists of a pair of web pages. The test passes if the pages are painted identically. Ideal for testing visual web features.
M make mochitest-plain Mochitest pages are loaded in the browser with low privileges. Ideal for thoroughly testing a web feature.
Moth make mochitest-chrome Chrome mochitest pages are loaded with high privileges.
make mochitest-browser-chrome Browser chrome tests run in a browser window's scope. Ideal for testing front-end code directly.
Z mozmill instructions Mozmill is a record/playback style automation tool. End-user interaction is captured as JS that calls an automation API.

With the exception of R and C, all individual tests define their own pass and fail conditions.

All test suites treat crashes, hangs, and (in debug builds) trace-refcnt leaks as failures. NS_ASSERTIONs are treated as fatal in X, annotatable failures in R/C/J, and ignored in M.

Most suites share a common test log format, so you can grep for UNEXPECTED to find failures in your run.

You can debug with EXTRA_TEST_ARGS='--debugger=gdb' make [suite].

Further Reading

Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.

There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and Automated testing tips and tricks if you're looking for something to read.

These are some additional efforts underway:

  • You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM.
    • {{ Bug("343673") }} tracks one person's effort, but there has been no recent activity
    • {{ Bug("346703") }} contains one example of how this might be done
  • JSUnit can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges.
    • JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.
    • {{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.
    • Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.
    • Some XForms tests uses JsUnit, as an example

Existing Test Harnesses and Frameworks

(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})

{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}

Revision Source

<div class="note"><strong>Note:</strong> It has been proposed that this page and <a href="/en/Developing_Tests" title="en/Developing tests">Developing tests</a> be merged.</div>
<p>This page gives an overview of options for automated testing available to Mozilla developers.</p>
<p>After <a href="/En/Simple_Firefox_build" title="En/Simple_Firefox_build">building Firefox</a> with --enable-tests, you can run any regression-test suite from your objdir with a simple make command.</p>
<p>Mozilla uses several test frameworks. They are listed here in rough order from the lowest-level unit tests to the highest-level system tests.</p>
<table border="1"> <thead> <tr> <th>TBPL<br> code</th> <th>Command</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>(B)</td> <td><code>make check</code></td> <td><a href="/en/Compiled-code_automated_tests" title="en/Compiled-code_automated_tests">Compiled-code tests</a> can test non-scriptable interfaces, but are hard to write and maintain.</td> </tr> <tr> <td>X</td> <td><code>make xpcshell-tests</code></td> <td>JavaScript code runs in <a href="/en/XPConnect/xpcshell" title="en/XPConnect/xpcshell">xpcshell</a>. Tests can access XPCOM components, but can't easily work with documents or windows.</td> </tr> <tr> <td>J</td> <td><code>make jstestbrowser</code></td> <td>JavaScript engine regression tests.</td> </tr> <tr> <td>C</td> <td><code>make crashtest</code></td> <td>A web page is loaded.</td> </tr> <tr> <td>R</td> <td><code>make reftest</code></td> <td>A <a href="/en/Creating_reftest-based_unit_tests" title="en/Creating_reftest-based_unit_tests">reftest</a> consists of a <em>pair</em> of web pages. The test passes if the pages are painted identically. Ideal for testing visual web features.</td> </tr> <tr> <td>M</td> <td><code>make mochitest-plain</code></td> <td><a href="/en/Mochitest" title="en/Mochitest">Mochitest pages</a> are loaded in the browser with low privileges. Ideal for thoroughly testing a web feature.</td> </tr> <tr> <td rowspan="2">Moth</td> <td><code>make mochitest-chrome</code></td> <td><a href="/en/Chrome_tests" title="en/Chrome_tests">Chrome mochitest pages</a> are loaded with high privileges.</td> </tr> <tr> <td><code>make mochitest-browser-chrome</code></td> <td><a href="/en/Browser_chrome_tests" title="en/Browser_chrome_tests">Browser chrome tests</a> run in a browser window's scope. Ideal for testing front-end code directly.</td> </tr> <tr> <td>Z</td> <td><a class=" link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=610941#c9">mozmill instructions</a></td> <td><a class="internal" href="/en/Mozmill_Tests" title="en/Mozmill Tests">Mozmill</a> is a record/playback style automation tool. End-user interaction is captured as JS that calls an automation API.</td> </tr> </tbody>
</table>
<p>With the exception of R and C, all individual tests define their own pass and fail conditions.</p>
<p>All test suites treat crashes, hangs, and (in debug builds) <a class=" link-https" href="https://wiki.mozilla.org/Performance:Leak_Tools">trace-refcnt leaks</a> as failures. <a href="/en/NS_ASSERTION" title="en/NS ASSERTION">NS_ASSERTION</a>s are treated as fatal in X, annotatable failures in R/C/J, and ignored in M.</p>
<p>Most suites share a common <a href="/en/Test_log_format" rel="internal">test log format</a>, so you can grep for UNEXPECTED to find failures in your run.</p>
<p>You can debug with <code>EXTRA_TEST_ARGS='--debugger=gdb' make [suite]</code>.</p>
<h3 name="Further_Reading">Further Reading</h3>
<p>Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.</p>
<p>There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and <a href="/en/Automated_testing_tips_and_tricks" title="en/Automated_testing_tips_and_tricks">Automated testing tips and tricks</a> if you're looking for something to read.</p>
<p>These are some additional efforts underway:</p>
<ul> <li>You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM. <ul> <li><em>{{ Bug("343673") }} tracks one person's effort, but there has been no recent activity</em></li> <li><em>{{ Bug("346703") }} contains one example of how this might be done</em></li> </ul> </li> <li><a class="external" href="http://www.jsunit.net/">JSUnit</a> can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges. <ul> <li>JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.</li> <li>{{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.</li> <li>Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.</li> <li>Some <a class="external" href="http://beaufour.dk/xftst/">XForms tests</a> uses JsUnit, as an example</li> </ul> </li>
</ul>
<h3 name="Existing_Test_Harnesses_and_Frameworks">Existing Test Harnesses and Frameworks</h3>
<p>(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})</p>
<ul> <li><a class="external" href="http://wiki.mozilla.org/Performance:Tinderbox_Tests">Tinderbox performance tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/browser/components/places/tests/">Places test script</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/netwerk/test/unit/">Netwerk unit tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/js/tests/">javascript tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/security/nss/tests/">nss tests</a></li> <li><a class="external" href="http://landfill.mozilla.org/mxr-test/mozilla/source/nsprpub/pr/tests/">nspr tests</a></li> <li><a class="external" href="http://www.mozilla.org/newlayout/doc/regression_tests.html">layout tests - diff of frame tree dump against golden master</a></li> <li><a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=301260">bz's copy of netwerk tests for xmlserializer</a></li> <li><a class="external" href="http://www.w3.org/DOM/Test/">W3C DOM Tests</a> uses <a class="external" href="http://www.edwardh.com/jsunit/">jsunit</a></li> <li><a class="external" href="http://wiki.mozilla.org/XSLT_Tests">XSLT Tests</a></li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST" rel="freelink">http://hixie.ch/tests/MANIFEST</a> all hixie's tests</li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST-visual" rel="freelink">http://hixie.ch/tests/MANIFEST-visual</a> subset of hixie's tests that is non-interactive</li> <li><a class="external" href="http://www.allpeers.com/blog/2005/09/28/foxunit-unit-test-framework-for-firefox/">FoxUnit</a> - jUnit-like harness for Firefox, by the AllPeers folks</li>
</ul>
<p>{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}</p>
Revert to this revision