Revision 5225 of Running automated tests

  • Revision slug: Running_automated_tests
  • Revision title: Running automated tests
  • Revision id: 5225
  • Created:
  • Creator: lunalum
  • Is current revision? No
  • Comment no wording changes

Revision Content

Mozilla code is covered by a large number of automated unit- and regression tests. These tests are run on Tinderbox constantly and developers are expected to make sure their changes do not break the automated test-suites.

Mozilla uses several home-grown automated testing frameworks, an overview of each framework with links to detailed documentation is available at Mozilla automated testing.

This page lists the steps to run the automated tests.

Configuring the build

In order to run most tests, you must have a properly configured build. Platform (Gecko, Toolkit) tests, as well as Firefox-specific tests, are usually run on a Firefox build.

Build Documentation has the general instructions on building Firefox. The default build options are suitable for running the automated tests. The tests can be run on both debug and release builds, but due to bugs in some tests, they may die because of an assertion failure. (XXX find/file a bug that tracks such tests.)

make check

The following commands execute the standalone (aka "compiled-code") tests and the xpcshell-based tests.

$ cd <OBJ_DIR>
$ make check

If any of the tests fail, you get a message like:

make: *** [check] Error 2

If make exits without an error, all the tests passed.

Mochitest

To run the whole Mochitest suite use the following commands:

cd <OBJ_DIR>
cd _tests/testing/mochitest
python runtests.py --autorun

The tests may take a while to complete (40 minutes on a 2Ghz Macbook as of June 2008) and the browser window must be focused during the test, otherwise some tests will fail.

See the individual sections on the Mochitest page for more information.

Reftests

Running a reftest requires creating a profile for reftest ({{ Bug("374458") }}). After that, use the following command to run reftests:

$ cd <SRC_DIR>
$ OBJ_DIR/dist/MinefieldDebug.app/Contents/MacOS/firefox -no-remote \
  -P PROFILE_NAME -reftest layout/reftests/reftest.list 2>&1 | grep UNEXPECTED

where

  • SRC_DIR is the top of the mozilla source tree (e.g. mozilla/),
  • OBJ_DIR is the folder with the build
  • PROFILE_NAME is the name of the profile you created for reftest
  • replace dist/MinefieldDebug.app/Contents/MacOS/firefox with the relative path to firefox executable, which on Linux and Windows is simply dist/bin/firefox

Reftests take about 20 minutes on the configuration listed above.

If the command prints any output (UNEXPECTED FAIL or similar), some reftests have failed.

Other tests not covered in this document

  • TODO: crashtest, chrome tests, ...
  • Tinderbox performance tests - Tp, Txul, Ts, etc.
  • Talos
  • running the above tests with/without leak, assertion checks

Revision Source

<p>Mozilla code is covered by a large number of automated unit- and regression tests. These tests are run on <a href="/en/Tinderbox" title="en/Tinderbox">Tinderbox</a> constantly and developers are expected to make sure their changes do not break the automated test-suites.</p>
<p>Mozilla uses several home-grown automated testing frameworks, an overview of each framework with links to detailed documentation is available at <a href="/en/Mozilla_automated_testing" title="en/Mozilla_automated_testing">Mozilla automated testing</a>.</p>
<p>This page lists the steps to run the automated tests.</p>
<h3 name="Configuring_the_build">Configuring the build</h3>
<p>In order to run most tests, you must have a properly configured build. Platform (Gecko, Toolkit) tests, as well as Firefox-specific tests, are usually run on a Firefox build.</p>
<p><a href="/en/Build_Documentation" title="en/Build_Documentation">Build Documentation</a> has the general instructions on building Firefox. The default build options are suitable for running the automated tests. The tests can be run on both debug and release builds, but due to bugs in some tests, they may die because of an assertion failure. (XXX find/file a bug that tracks such tests.)</p>
<h3 name="make_check"><code>make check</code></h3>
<p>The following commands execute the <a href="/en/Compiled-code_automated_tests" title="en/Compiled-code_automated_tests">standalone (aka "compiled-code") tests</a> and the <a href="/en/Writing_xpcshell-based_unit_tests" title="en/Writing_xpcshell-based_unit_tests">xpcshell-based tests</a>.</p>
<pre class="eval">$ cd <strong>&lt;OBJ_DIR&gt;</strong>
$ make check
</pre>
<p>If any of the tests fail, you get a message like:</p>
<pre class="eval">make: *** [check] Error 2
</pre>
<p>If <code>make</code> exits without an error, all the tests passed.</p>
<h3 name="Mochitest">Mochitest</h3>
<p>To run the whole <a href="/en/Mochitest" title="en/Mochitest">Mochitest</a> suite use the following commands:</p>
<pre class="eval">cd <strong>&lt;OBJ_DIR&gt;</strong>
cd _tests/testing/mochitest
python runtests.py --autorun
</pre>
<p>The tests may take a while to complete (40 minutes on a 2Ghz Macbook as of June 2008) and the browser window must be focused during the test, otherwise some tests will fail.</p>
<p>See the individual sections on the <a href="/en/Mochitest" title="en/Mochitest">Mochitest</a> page for more information.</p>
<h3 name="Reftests">Reftests</h3>
<p>Running a reftest requires creating a profile for reftest ({{ Bug("374458") }}). After that, use the following command to run reftests:</p>
<pre class="eval">$ cd <strong>&lt;SRC_DIR&gt;</strong>
$ <strong>OBJ_DIR</strong>/dist/MinefieldDebug.app/Contents/MacOS/firefox -no-remote \
  -P <strong>PROFILE_NAME</strong> -reftest layout/reftests/reftest.list 2&gt;&amp;1 | grep UNEXPECTED
</pre>
<p>where</p>
<ul> <li>SRC_DIR is the top of the mozilla source tree (e.g. <code>mozilla/</code>),</li> <li>OBJ_DIR is the folder with the build</li> <li>PROFILE_NAME is the name of the profile you created for reftest</li> <li>replace <code>dist/MinefieldDebug.app/Contents/MacOS/firefox</code> with the relative path to firefox executable, which on Linux and Windows is simply <code>dist/bin/firefox</code></li>
</ul>
<p>Reftests take about 20 minutes on the configuration listed above.</p>
<p>If the command prints any output (<code>UNEXPECTED FAIL</code> or similar), some reftests have failed.</p>
<h3 name="Other_tests_not_covered_in_this_document">Other tests not covered in this document</h3>
<ul> <li>TODO: crashtest, <a href="/en/Chrome_tests" title="en/Chrome_tests">chrome tests</a>, ...</li> <li>Tinderbox performance tests - Tp, Txul, Ts, etc. <a class="external" href="http://wiki.mozilla.org/Performance:Tinderbox_Tests"></a></li> <li>Talos</li> <li>running the above tests with/without leak, assertion checks</li>
</ul>
Revert to this revision