Mozilla automated testing

  • Revision slug: Mozilla_automated_testing
  • Revision title: Mozilla automated testing
  • Revision id: 4161
  • Created:
  • Creator: Jorend
  • Is current revision? No
  • Comment 38 words added, 71 words removed

Revision Content

Note: It has been proposed that this page and Developing tests be merged.

This page gives an overview of options for automated testing available to Mozilla developers.

Most automated tests should be executed on make check. How to add a build-time test describes the steps required to add an arbitrary test program to the test suite. Depending on what you need to test, you can use one of the testing frameworks available:

Unit testing

xpcshell: make xpcshell-tests

With xpcshell test harness you write unit tests in JavaScript. The code later runs in xpcshell, which is an XPConnect-enabled JS shell. This means your code can access XPCOM components, but can't (easily) open windows, test the application's chrome, work with HTML parser or the DOM.

A simple HTTP server is available for use in xpcshell tests.

Compiled-code tests: make check

Unscriptable and similar functionality requires writing C++ tests to exercise the functionality. Despite appearances, very few tests of this nature exist in the tree at the moment. There are many tests of this nature in the tree (see the contents of {{ Source("xpcom/tests") }} for many examples), but many can only be run manually, most report failure using customized messages printed to the console (and thus can't be automatically checked to have run correctly), and many fail to compile or contain numerous bugs.

Nevertheless, some C++ tests have been written such that they can be run automatically and detect failures. The current best example in terms of ease-of-authoring is {{ Source("xpcom/tests/TestPipe.cpp") }}, which uses a special test-support header intended to make writing such tests easy. See Compiled-code automated tests for details.

Writing a C++ test is more difficult than writing an xpcshell test, and the code is harder to maintain and modify. Don't write a compiled-code test if you don't have to! You should use these tests only when the functionality being tested depends upon unscriptable interfaces, methods, or properties, as much as possible.

Graphical and/or interactive tests

Mochitest

Mochitest is a framework based on Mochikit for writing tests. Tests run in the browser, from a local web server (provided by Mochitest). The start scripts that accompany it grant several privileges to the localhost (it creates a fresh profile each run). As a result, unit tests are free to request UniversalXPConnect privileges (access XPCOM components), open popup windows, etc. Mochitest is a good fit if you really need a full browser to test a problem. For example, a recent test verifies that form submission works in iframes set to display:none. There's a Mochitest FAQ.

To run these tests, use make mochitest-plain

Chrome tests

Chrome tests are meant to be used to test "chrome" code, and any other features testable from chrome JavaScript code.

To run these tests, use make mochitest-chrome.

Browser chrome tests

Browser chrome tests are meant to be used to test front-end "chrome" code, and any other features testable from chrome JavaScript code. They consist of JavaScript code snippets that are run in the browser window's scope.

To run these tests, use make mochitest-browser-chrome.

Reftest

{{ Source("layout/tools/reftest/README.txt", "Layout Engine Visual Tests (reftest)") }}. Each test typically consists of two files - one of them contains test markup, and the other typically contains reference markup (although using a static PNG is also possible if desired). The system works by comparing the rendering of the two files.

reftest is now built and installed automatically when ENABLE_TESTS is on (default) and there are example tests in {{ Source("layout/reftests") }}.

To run all the layout reftests, use make reftest. ("make lcheck" is supposed to do that for you, but it's currently broken.) You can grep the results for UNEXPECTED or run one of the reftest output processing tools listed in the readme.

 

To run crashtests, which also use the reftest harness, use make crashtest.

And to run thousands of JavaScript regression tests in the browser, also using the reftest harness, use make jstestbrowser.

For more information, see {{ Source("layout/tools/reftest/README.txt", "the reftest documentation") }} and the tutorial on creating a reftest-based test.

Crash tests

The crash test framework is based on the reftest framework, so writing and executing tests is much the same. Each test is a single file and the test works by loading the file and completing without crashing or asserting (the framework does not explicitly watch for asserts at this time, you need to grep for them in the output).

To run all the crash tests, from the object directory type

make crashtest

Mozmill tests

Mozmill is a record/playback style automation tool for automating end user interaction with an application.  It can run on any application developed from Gecko 1.9.0 and later.  Tests are written in simple JavaScript against an automation API.

Further Reading

Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.

There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and Automated testing tips and tricks if you're looking for something to read.

These are some additional efforts underway:

  • You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM.
    • {{ Bug("343673") }} tracks one person's effort, but there has been no recent activity
    • {{ Bug("346703") }} contains one example of how this might be done
  • JSUnit can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges.
    • JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.
    • {{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.
    • Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.
    • Some XForms tests uses JsUnit, as an example

Existing Test Harnesses and Frameworks

(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})

{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}

Revision Source

<div class="note"><strong>Note:</strong> It has been proposed that this page and <a href="/en/Developing_Tests" title="en/Developing tests">Developing tests</a> be merged.</div>
<p>This page gives an overview of options for automated testing available to Mozilla developers.</p>
<p>Most automated tests should be executed on <code>make check</code>. <a href="/en/How_to_add_a_build-time_test" title="en/How_to_add_a_build-time_test">How to add a build-time test</a> describes the steps required to add an arbitrary test program to the test suite. Depending on what you need to test, you can use one of the testing frameworks available:</p>
<h3 name="Unit_testing">Unit testing</h3>
<h4 name="xpcshell:_make_check">xpcshell: make xpcshell-tests</h4>
<p>With <a href="/en/Writing_xpcshell-based_unit_tests" title="en/Writing_xpcshell-based_unit_tests">xpcshell test harness</a> you write unit tests in JavaScript. The code later runs in <a href="/en/XPConnect/xpcshell" title="en/xpcshell">xpcshell</a>, which is an <a href="/en/XPConnect" title="en/XPConnect">XPConnect</a>-enabled JS shell. This means your code can access XPCOM components, but can't (easily) open windows, test the application's chrome, work with HTML parser or the DOM.</p>
<p>A simple <a href="/En/Httpd.js/HTTP_server_for_unit_tests" title="en/HTTP_server_for_unit_tests">HTTP server</a> is available for use in xpcshell tests.</p>
<h4 name="Compiled-code_tests">Compiled-code tests: make check</h4>
<p>Unscriptable and similar functionality requires writing C++ tests to exercise the functionality. Despite appearances, very few tests of this nature exist in the tree at the moment. There are many tests of this nature in the tree (see the contents of {{ Source("xpcom/tests") }} for many examples), but many can only be run manually, most report failure using customized messages printed to the console (and thus can't be automatically checked to have run correctly), and many fail to compile or contain numerous bugs.</p>
<p>Nevertheless, some C++ tests have been written such that they can be run automatically and detect failures. The current best example in terms of ease-of-authoring is {{ Source("xpcom/tests/TestPipe.cpp") }}, which uses a special test-support header intended to make writing such tests easy. See <a href="/en/Compiled-code_automated_tests" title="en/Compiled-code_automated_tests">Compiled-code automated tests</a> for details.</p>
<p>Writing a C++ test is more difficult than writing an xpcshell test, and the code is harder to maintain and modify. <strong>Don't write a compiled-code test if you don't have to!</strong> You should use these tests only when the functionality being tested depends upon unscriptable interfaces, methods, or properties, as much as possible.</p>
<h3 name="Graphical_and.2For_interactive_tests">Graphical and/or interactive tests</h3>
<h4 name="Mochitest">Mochitest</h4>
<p><a href="/en/Mochitest" title="en/Mochitest">Mochitest</a> is a framework based on <a class="external" href="http://mochikit.com/">Mochikit</a> for writing tests. Tests run in the browser, from a local web server (provided by Mochitest). The start scripts that accompany it grant several privileges to the localhost (it creates a fresh profile each run). As a result, unit tests are free to request UniversalXPConnect privileges (access XPCOM components), open popup windows, etc. Mochitest is a good fit if you really need a full browser to test a problem. For example, a recent test verifies that form submission works in iframes set to <code>display:none</code>. There's a <a href="/en/Mochitest#FAQ" title="en/Mochitest#FAQ">Mochitest FAQ</a>.</p>
<p>To run these tests, use <code>make mochitest-plain</code></p>
<h4 name="Chrome_tests">Chrome tests</h4>
<p><a href="/en/Chrome_tests" title="en/Chrome_tests">Chrome tests</a> are meant to be used to test "chrome" code, and any other features testable from chrome JavaScript code.</p>
<p>To run these tests, use <code>make mochitest-chrome</code>.</p>
<h4 name="Browser_chrome_tests">Browser chrome tests</h4>
<p><a href="/en/Browser_chrome_tests" title="en/Browser_chrome_tests">Browser chrome tests</a> are meant to be used to test front-end "chrome" code, and any other features testable from chrome JavaScript code. They consist of JavaScript code snippets that are run in the browser window's scope.</p>
<p>To run these tests, use <code>make mochitest-browser-chrome</code>.</p>
<h4 name="Reftest">Reftest</h4>
<p>{{ Source("layout/tools/reftest/README.txt", "Layout Engine Visual Tests (reftest)") }}. Each test typically consists of two files - one of them contains test markup, and the other typically contains reference markup (although using a static PNG is also possible if desired). The system works by comparing the rendering of the two files.</p>
<p>reftest is now built and installed automatically when <code>ENABLE_TESTS</code> is on (default) and there are example tests in {{ Source("layout/reftests") }}.</p>
<p>To run all the layout reftests, use <code>make reftest</code>. ("make lcheck" is supposed to do that for you, but it's currently broken.) You can grep the results for <code style="color: rgb(37, 34, 29); font-weight: inherit; ">UNEXPECTED</code> or run one of the reftest output processing tools listed in the readme.</p>
<p> </p>
<p>To run crashtests, which also use the reftest harness, use <code>make crashtest</code>.</p>
<p>And to run thousands of JavaScript regression tests in the browser, also using the reftest harness, use <code>make jstestbrowser</code>.</p>
<p>For more information, see {{ Source("layout/tools/reftest/README.txt", "the reftest documentation") }} and <a href="/en/Creating_reftest-based_unit_tests" title="en/Creating_reftest-based_unit_tests">the tutorial on creating a reftest-based test</a>.</p><h4 name="Crash_tests">Crash tests</h4>
<p>The crash test framework is based on the reftest framework, so writing and executing tests is much the same. Each test is a single file and the test works by loading the file and completing without crashing or asserting (the framework does not explicitly watch for asserts at this time, you need to grep for them in the output).</p>
<p>To run all the crash tests, from the object directory type</p>
<p>make crashtest</p>
<h4 name="Crash_tests">Mozmill tests</h4>
<p><a class="internal" href="/en/Mozmill_Tests" title="en/Mozmill Tests">Mozmill</a> is a record/playback style automation tool for automating end user interaction with an application.  It can run on any application developed from Gecko 1.9.0 and later.  Tests are written in simple JavaScript against an automation API.</p>
<h3 name="Further_Reading">Further Reading</h3>
<p>Please ignore the {{ interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad') }} page, and look at {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.</p>
<p>There's also {{ interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting') }} and <a href="/en/Automated_testing_tips_and_tricks" title="en/Automated_testing_tips_and_tricks">Automated testing tips and tricks</a> if you're looking for something to read.</p>
<p>These are some additional efforts underway:</p>
<ul> <li>You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM. <ul> <li><em>{{ Bug("343673") }} tracks one person's effort, but there has been no recent activity</em></li> <li><em>{{ Bug("346703") }} contains one example of how this might be done</em></li> </ul> </li> <li><a class="external" href="http://www.jsunit.net/">JSUnit</a> can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges. <ul> <li>JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.</li> <li>{{ interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect') }} lists some jsunit examples.</li> <li>Refer to documentation on {{ interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit') }} for more information.</li> <li>Some <a class="external" href="http://beaufour.dk/xftst/">XForms tests</a> uses JsUnit, as an example</li> </ul> </li>
</ul>
<h3 name="Existing_Test_Harnesses_and_Frameworks">Existing Test Harnesses and Frameworks</h3>
<p>(originally from {{ interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests') }})</p>
<ul> <li><a class="external" href="http://wiki.mozilla.org/Performance:Tinderbox_Tests">Tinderbox performance tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/browser/components/places/tests/">Places test script</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/netwerk/test/unit/">Netwerk unit tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/js/tests/">javascript tests</a></li> <li><a class="external" href="http://lxr.mozilla.org/mozilla/source/security/nss/tests/">nss tests</a></li> <li><a class="external" href="http://landfill.mozilla.org/mxr-test/mozilla/source/nsprpub/pr/tests/">nspr tests</a></li> <li><a class="external" href="http://www.mozilla.org/newlayout/doc/regression_tests.html">layout tests - diff of frame tree dump against golden master</a></li> <li><a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=301260">bz's copy of netwerk tests for xmlserializer</a></li> <li><a class="external" href="http://www.w3.org/DOM/Test/">W3C DOM Tests</a> uses <a class="external" href="http://www.edwardh.com/jsunit/">jsunit</a></li> <li><a class="external" href="http://wiki.mozilla.org/XSLT_Tests">XSLT Tests</a></li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST" rel="freelink">http://hixie.ch/tests/MANIFEST</a> all hixie's tests</li> <li><a class=" external" href="http://hixie.ch/tests/MANIFEST-visual" rel="freelink">http://hixie.ch/tests/MANIFEST-visual</a> subset of hixie's tests that is non-interactive</li> <li><a class="external" href="http://www.allpeers.com/blog/2005/09/28/foxunit-unit-test-framework-for-firefox/">FoxUnit</a> - jUnit-like harness for Firefox, by the AllPeers folks</li>
</ul>
<p>{{ languages( { "es": "es/Pruebas_automatizadas_de_Mozilla", "ja": "Ja/Mozilla_automated_testing" } ) }}</p>
Revert to this revision