Mozilla automated testing

  • Revision slug: Mozilla_automated_testing
  • Revision title: Mozilla automated testing
  • Revision id: 4142
  • Created:
  • Creator: Waldo
  • Is current revision? No
  • Comment compiled-code tests

Revision Content

When finished, this page will give an overview of options for automated testing available to Mozilla developers with links to more documentation.

Most automated tests should be executed on make check. How to add a build-time test describes the steps required to add an arbitrary test program to the test suite. Depending on what you need to test, you can use one of the testing frameworks available:

Unit testing

xpcshell: make check

With xpcshell test harness you write unit tests in JavaScript. The code later runs in xpcshell, which is an XPConnect-enabled JS shell. This means your code can access XPCOM components, but can't (easily) open windows, test the application's chrome, work with HTML parser or the DOM.

A simple HTTP server is available for use in xpcshell tests.

Compiled-code tests

Unscriptable and similar functionality requires writing C++ tests to exercise the functionality. Despite appearances, very few tests of this nature exist in the tree at the moment. There are many tests of this nature in the tree (see the contents of {{template.Source("xpcom/tests")}} for many examples), but many can only be run manually, most report failure using customized messages printed to the console (and thus can't be automatically checked to have run correctly), and many fail to compile or contain numerous bugs.

Nevertheless, some C++ tests have been written such that they can be run automatically and detect failures. The current best example in terms of ease-of-authoring is {{template.Source("xpcom/tests/TestPipe.cpp")}}, which uses a special test-support header intended to make writing such tests easy. See Compiled-code automated tests for details.

Writing a C++ test is more difficult than writing an xpcshell test, and the code is harder to maintain and modify. Don't write a compiled-code test if you don't have to! You should use these tests only when the functionality being tested depends upon unscriptable interfaces, methods, or properties, as much as possible.

Graphical and/or interactive tests

Mochitest

Mochitest is a framework based on Mochikit for writing tests. Tests run in the browser, from a local web server (provided by Mochitest). The start scripts that accompany it grant several privileges to the localhost (it creates a fresh profile each run). As a result, unit tests are free to request UniversalXPConnect privileges (access XPCOM components), open popup windows, etc. Mochitest is a good fit if you really need a full browser to test a problem. For example, a recent test verifies that form submission works in iframes set to display:none. There's a Mochitest FAQ.

Browser chrome tests

Browser chrome tests are meant to be used to test front-end "chrome" code, and any other features testable from chrome JavaScript code. They consist of JavaScript code snippets that are run in the browser window's scope.

Reftest

{{template.Source("layout/tools/reftest/README.txt", "Layout Engine Visual Tests (reftest)")}}. Each test consists of two documents (e.g. HTML) - one of them containing test markup and the other containing reference markup. The system works by comparing the rendering of two documents.

reftest is now built and installed automatically when ENABLE_TESTS is on (default) and there are example tests in {{template.Source("layout/reftests")}}.

To run all the reftests, type

obj-debug/dist/MinefieldDebug.app/Contents/MacOS/firefox -reftest layout/reftests/reftest.list

("make lcheck" is supposed ot do that for you, but it's currently broken.)

You can grep the results for "UNEXPECTED" or run one of the reftest output processing tools listed in the readme.

You may want to use a separate profile to run the tests (by creating a profile named "reftests" and adding -P reftests to the line invoking the application).

For more information, see {{template.Source("layout/tools/reftest/README.txt", "")}} and the tutorial on creating a reftest-based test.

Further Reading

Please ignore the {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad')}} page, and look at {{mediawiki.interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting')}} only. The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.

There's also {{mediawiki.interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting')}} and Automated testing tips and tricks if you're looking for something to read.

These are some additional efforts underway:

  • You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM.
    • {{template.Bug(343673)}} tracks one person's effort, but there has been no recent activity
    • {{template.Bug(346703)}} contains one example of how this might be done
  • JSUnit can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges.
    • JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.
    • {{mediawiki.interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect')}} lists some jsunit examples.
    • Refer to documentation on {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit')}} for more information.
    • Some XForms tests uses JsUnit, as an example

Existing Test Harnesses and Frameworks

(originally from {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests')}})

{{ wiki.languages( { "es": "es/Pruebas_automatizadas_de_Mozilla" } ) }}

Revision Source

<p>
</p><p>When finished, this page will give an overview of options for automated testing available to Mozilla developers with links to more documentation.
</p><p>Most automated tests should be executed on <code>make check</code>. <a href="en/How_to_add_a_build-time_test">How to add a build-time test</a> describes the steps required to add an arbitrary test program to the test suite. Depending on what you need to test, you can use one of the testing frameworks available:
</p>
<h3 name="Unit_testing"> Unit testing </h3>
<h4 name="xpcshell:_make_check"> xpcshell: make check </h4>
<p>With <a href="en/Writing_xpcshell-based_unit_tests">xpcshell test harness</a> you write unit tests in JavaScript. The code later runs in <a href="en/Xpcshell">xpcshell</a>, which is an <a href="en/XPConnect">XPConnect</a>-enabled JS shell. This means your code can access XPCOM components, but can't (easily) open windows, test the application's chrome, work with HTML parser or the DOM.
</p><p>A simple <a href="en/HTTP_server_for_unit_tests">HTTP server</a> is available for use in xpcshell tests.
</p>
<h4 name="Compiled-code_tests"> Compiled-code tests </h4>
<p>Unscriptable and similar functionality requires writing C++ tests to exercise the functionality.  Despite appearances, very few tests of this nature exist in the tree at the moment.  There are many tests of this nature in the tree (see the contents of {{template.Source("xpcom/tests")}} for many examples), but many can only be run manually, most report failure using customized messages printed to the console (and thus can't be automatically checked to have run correctly), and many fail to compile or contain numerous bugs.
</p><p>Nevertheless, some C++ tests have been written such that they can be run automatically and detect failures.  The current best example in terms of ease-of-authoring is {{template.Source("xpcom/tests/TestPipe.cpp")}}, which uses a special test-support header intended to make writing such tests easy.  See <a href="en/Compiled-code_automated_tests">Compiled-code automated tests</a> for details.
</p><p>Writing a C++ test is more difficult than writing an xpcshell test, and the code is harder to maintain and modify.  <b>Don't write a compiled-code test if you don't have to!</b>  You should use these tests only when the functionality being tested depends upon unscriptable interfaces, methods, or properties, as much as possible.
</p>
<h3 name="Graphical_and.2For_interactive_tests"> Graphical and/or interactive tests </h3>
<h4 name="Mochitest"> Mochitest </h4>
<p><a href="en/Mochitest">Mochitest</a> is a framework based on <a class="external" href="http://mochikit.com/">Mochikit</a> for writing tests. Tests run in the browser, from a local web server (provided by Mochitest). The start scripts that accompany it grant several privileges to the localhost (it creates a fresh profile each run). As a result, unit tests are free to request UniversalXPConnect privileges (access XPCOM components), open popup windows, etc. Mochitest is a good fit if you really need a full browser to test a problem. For example, a recent test verifies that form submission works in iframes set to <code>display:none</code>. There's a <a href="en/Mochitest#FAQ">Mochitest FAQ</a>.
</p>
<h4 name="Browser_chrome_tests"> Browser chrome tests </h4>
<p><a href="en/Browser_chrome_tests">Browser chrome tests</a> are meant to be used to test front-end "chrome" code, and any other features testable from chrome JavaScript code. They consist of JavaScript code snippets that are run in the browser window's scope.
</p>
<h4 name="Reftest"> Reftest </h4>
<p>{{template.Source("layout/tools/reftest/README.txt", "Layout Engine Visual Tests (reftest)")}}. Each test consists of two documents (e.g. HTML) - one of them containing test markup and the other containing reference markup. The system works by comparing the rendering of two documents.
</p><p>reftest is now built and installed automatically when <code>ENABLE_TESTS</code> is on (default) and there are example tests in {{template.Source("layout/reftests")}}.
</p><p>To run all the reftests, type
</p><p><code>obj-debug/dist/MinefieldDebug.app/Contents/MacOS/firefox -reftest layout/reftests/reftest.list</code>
</p><p>("make lcheck" is supposed ot do that for you, but it's currently broken.)
</p><p>You can grep the results for "UNEXPECTED" or run one of the reftest output processing tools listed in the readme.
</p><p>You may want to use a separate profile to run the tests (by creating a profile named "reftests" and adding <code>-P reftests</code> to the line invoking the application).
</p><p>For more information, see {{template.Source("layout/tools/reftest/README.txt", "")}} and <a href="en/Creating_reftest-based_unit_tests">the tutorial on creating a reftest-based test</a>.
</p>
<h3 name="Further_Reading"> Further Reading </h3>
<p>Please ignore the {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Scratchpad', 'wikimo:SoftwareTesting:Scratchpad')}} page, and look at {{mediawiki.interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting')}} only.  The scratchpad is for works-in-progress, and is pretty much guaranteed to be stale or wrong.
</p><p>There's also {{mediawiki.interwiki('wikimo', 'SoftwareTesting', 'wikimo:SoftwareTesting')}} and <a href="en/Automated_testing_tips_and_tricks">Automated testing tips and tricks</a> if you're looking for something to read.
</p><p>These are some additional efforts underway:
</p>
<ul><li> You can write standalone test programs in C/C++. This option can be used to test functionality not exposed via XPCOM. 
<ul><li> <i>{{template.Bug(343673)}} tracks one person's effort, but there has been no recent activity</i>
</li><li> <i>{{template.Bug(346703)}} contains one example of how this might be done</i>
</li></ul>
</li><li> <a class="external" href="http://www.jsunit.net/">JSUnit</a> can be used to write tests that run as content in the browser. Particularly useful for DOM, parser tests, but can't do anything that requires chrome privileges. 
<ul><li> JsUnit will most likely not be a make check target for the time being, since it needs a full browser instance.
</li><li> {{mediawiki.interwiki('wikimo', 'SoftwareTesting#Ideas to Collect', 'wikimo:SoftwareTesting#Ideas_to_Collect')}} lists some jsunit examples.
</li><li> Refer to documentation on {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Tools:jsUnit', 'wikimo:SoftwareTesting:Tools:jsUnit')}} for more information.
</li><li> Some <a class="external" href="http://beaufour.dk/xftst/">XForms tests</a> uses JsUnit, as an example
</li></ul>
</li></ul>
<h3 name="Existing_Test_Harnesses_and_Frameworks"> Existing Test Harnesses and Frameworks </h3>
<p>(originally from {{mediawiki.interwiki('wikimo', 'SoftwareTesting:Catalog_of_Automated_Tests', 'wikimo:SoftwareTesting:Catalog_of_Automated_Tests')}})
</p>
<ul><li> <a class="external" href="http://wiki.mozilla.org/Performance:Tinderbox_Tests">Tinderbox performance tests</a>
</li><li> <a class="external" href="http://lxr.mozilla.org/mozilla/source/browser/components/places/tests/">Places test script</a>
</li><li> <a class="external" href="http://lxr.mozilla.org/mozilla/source/netwerk/test/unit/">Netwerk unit tests</a>
</li><li> <a class="external" href="http://lxr.mozilla.org/mozilla/source/js/tests/">javascript tests</a>
</li><li> <a class="external" href="http://lxr.mozilla.org/mozilla/source/security/nss/tests/">nss tests</a>
</li><li> <a class="external" href="http://landfill.mozilla.org/mxr-test/mozilla/source/nsprpub/pr/tests/">nspr tests</a>
</li><li> <a class="external" href="http://www.mozilla.org/newlayout/doc/regression_tests.html">layout tests - diff of frame tree dump against golden master</a>
</li><li> <a class="external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=301260">bz's copy of netwerk tests for xmlserializer</a>
</li><li> <a class="external" href="http://www.w3.org/DOM/Test/">W3C DOM Tests</a> uses <a class="external" href="http://www.edwardh.com/jsunit/">jsunit</a>
</li><li> <a class="external" href="http://wiki.mozilla.org/XSLT_Tests">XSLT Tests</a>
</li><li> http://hixie.ch/tests/MANIFEST all hixie's tests
</li><li> http://hixie.ch/tests/MANIFEST-visual subset of hixie's tests that is non-interactive
</li><li> <a class="external" href="http://www.allpeers.com/blog/2005/09/28/foxunit-unit-test-framework-for-firefox/">FoxUnit</a> - jUnit-like harness for Firefox, by the AllPeers folks
</li></ul>
{{ wiki.languages( { "es": "es/Pruebas_automatizadas_de_Mozilla" } ) }}
Revert to this revision