Writing xpcshell-based unit tests

  • Revision slug: Writing_xpcshell-based_unit_tests
  • Revision title: Writing xpcshell-based unit tests
  • Revision id: 21590
  • Created:
  • Creator: Mnyromyr
  • Is current revision? No
  • Comment /* Via <code>check-interactive</code> */

Revision Content

The xpcshell tool can be used to test certain kinds of functionality. Anything available to the XPCOM layer (through scriptable interfaces) can be tested with xpcshell. See Mozilla automated testing and category for pointers to more information.

Your first xpcshell-based test

Creating your first xpcshell-based test is easy. Create a file named <tt>test_first.js</tt> (unit test filenames must start <tt>test_</tt>) with the following:

function test_functionality_1()
{
  // do some complex action, return whether the action's results were what are expected here
  // this is just an example, so we return true
  return true;
}

function run_test()
{
  do_check_true(test_functionality_1());
}

This doesn't really test anything, but it gives you an idea how you'd actually write a test. If you want to execute the test, you must create a non-static build of the browser without --disable-tests. Add the test file to an existing set of tests (say, for example, {{ Source("tools/test-harness/xpcshell-simple/example/unit/") }}, but you'll want a different location when your test is posted for a review). You then execute your test using make check:

% /bin/sh
$ cd objdir
$ make -C tools/test-harness/xpcshell-simple/example check
...failure or success messages are printed to the console...

If you just want to run one test, directly, without any build system support, you can do this with something like the following commands. Suppose one wants to run "test_foo.js".

$ /bin/sh
$ export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"
$ export TOPSRCDIR="c:/path/to/top/srcdir"
$ cd dist/bin
$ ./run-mozilla.sh ./xpcshell -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js \
    -f test_foo.js \
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js
...failure or success messages are printed to the console...

To add arbitrary (not necessarily xpcshell-based) tests and to get assistance with testing in general, follow the instructions at How to add a build-time test.

{{ Fx_minversion_inline("3") }} Note:

You can now specify check-one instead of check on the command line to launch a single test ({{ Bug("404513") }}).

$ cd objdir
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-one

Adding to your test

The test is executed by the test harness by calling the run_test function defined in the file. Anything you want to test must be spawned from this function.

If you have a group of files in a directory and they all include some common code that needs to run before your tests, to set up resources for example, you can put one copy of that code in a source file whose name starts with head_ and ends with .js. Any code that is common to your test and which needs to run after your tests can be put into a source file that starts with tail_ and ends with .js. You should also name your test source files with names that start with test_ and end with .js. Each of your test files needs to contain at least a "run_test()" function. If the "run_test()" function runs to completion without throwing an exception, the test succeeds. (If you need to do any asynchronous tests, see do_test_pending and do_test_finished below.)

xpcshell tests have access to the following utility functions:

do_throw(messageText)
Call this function to report an error and exit the test. The argument is a string that will be reported in the test's log file.
Note: do_throw can not be caught by a try/catch block, the test will always exit.
do_check_eq(a, b)
Call this function to assert that two objects are equal. If not equal, an exception is logged and the test case is halted.
do_check_neq(a, b)
Call this function to assert that two objects are not equal. If equal, an exception is logged and the test case is halted.
do_check_true(expr)
Call this function to assert that expr is equal to true.
do_check_false(expr)
Call this function to assert that expr is equal to false.
do_timeout(delay, expr)
Call this function to schedule a timeout. The given string expression will be evaluated after the specified delay (in milliseconds). Note that you must call do_test_pending so that the test isn't completed before your timer fires, and you must call do_test_finished when the actions you perform in the timeout complete, if you have no other functionality to test.)
do_test_pending()
Delay exit of the test until do_test_finished() is called. do_test_pending() may be called multiple times, and do_test_finished() must be paired with each before the unit test will exit.
do_test_finished()
Call this function to inform the test framework that an asynchronous operation has completed. If all asynchronous operations have completed (i.e. every do_test_pending() has been matched with a do_test_finished() in execution), then the unit test will exit.
do_import_script(topsrcdirRelativePath)
Imports the JavaScript file referenced by topsrcdirRelativePath into the global script context, executing the code inside it. The file specified is a file within the source tree. For example, to use it with the tree's unit-testing HTTP server, simply use do_import_script("netwerk/test/httpserver/httpd.js"). Currently this function is only used with the HTTP server, although it could be used for any script guaranteed to be pulled with the code for which the tests are written.
do_get_file(topsrcdirRelativePath)
Returns an nsILocalFile object representing the given file (or directory) in the Mozilla tree, i.e. if topsrcdirRelativePath equals netwerk/test/TestServ.js, the returned file represents mozilla/netwerk/test/TestServ.js. The given path must be delimited with forward slashes; it is suggested that you end it with a forward slash if the file you are retrieving is a directory. You can use this to access test-specific auxiliary files if your test requires access to external files. Note also that you can use this to get directories; if you're accessing multiple files in the same location (e.g. foo/bar/baz, foo/bar/quux, and foo/bar/eit, it's probably worth getting the parent directory (in this case, foo/bar/) and cloning it to get each individual file.

Adding your tests to the list of mozilla tests

You should store your tests near the source of code you want to test. For example, imagine you want to add tests for your component in extensions/myextension/mycomponent, you will add a tests/ directory. You can store different kind of tests here (reftests etc.), and for xpcshell, you should create a unit directory and store all your javascript files inside it. Example: extensions/myextension/mycomponent/tests/unit/test_foo.js. The unit name is important because for the moment the xpcshell unit tests framework recognize only this name (especially when you want to launch a single test with check-interactive).

Then you should create a extensions/myextension/mycomponent/tests/Makefile.in which will contain XPCSHELL_TESTS = unit. Of course, in the extensions/myextension/mycomponent/Makefile.in you will add tests/ into the DIRS variable.

Running unit tests

Tests should be run using make -C OBJDIR/path_to_tests check. Just cd to an appropriate directory (e.g. to run tests for a single module, go to MOZ_OBJDIR/path_to_tests/test) and run make check.

To run a single test, use the check-one target, as described above.

Note: You must run make -C OBJDIR/path_to_tests before trying to run tests; the tests are copied to a central location to be executed, and this copying occurs in the libs target of the build, which make check does not invoke. (this note seems to be deprecated for Firefox3 sources)

Log files end up in MOZ_OBJDIR/_tests/xpcshell-simple/module_name/tests_type/*.log. PASS/FAIL information for each test, as well as for the test suite as a whole, is printed to the console. It's also included in Tinderbox logs on tinderboxes which have --enable-tests specified, so if you break a test you can see any error messages it printed in the build log.

Running unit tests under a C++ debugger

Via check-interactive

You can specify check-interactive when issuing the make command. This will cause your test to stop right before running so you can attach to it in a debugger (implemented in {{ Bug("382682") }}).

Example:

$ cd objdir
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-interactive
# js>_execute_test();
...failure or success messages are printed to the console...
# js>quit();

Note: For comm-central builds (only?), the path passed to -C, giving the relative path from the base source directory to the test directory, must not contain the final /unit/ subdirectory, even if the actual SOLO_FILE is stored there!

$ ls -la uriloader/exthandler/tests/unit/test_punycodeURIs.js
$ make SOLO_FILE="test_punycodeURIs.js" -C uriloader/exthandler/tests check-interactive

Running the test directly under gdb

Getting your test script running in gdb is fairly simple. First, you have to export two variables like so:

export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"
export TOPSRCDIR="c:/path/to/top/srcdir"

Then, run OBJDIR/dist/bin/run_mozilla.sh -g xpcshell, set your breakpoints or whatever else you need to do, then type run -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js -f path/to/test_file.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js. If you have head files that are automatically included, you should also include those before you include your test file with the -f option.

Revision Source

<p>
</p><p>The <a href="en/Xpcshell">xpcshell</a> tool can be used to test certain kinds of functionality. Anything available to the XPCOM layer (through scriptable interfaces) can be tested with xpcshell. See <a href="en/Mozilla_automated_testing">Mozilla automated testing</a> and <a href="Special:Tags?tag=Automated_testing&amp;language=en">category</a> for pointers to more information.
</p>
<h3 name="Your_first_xpcshell-based_test"> Your first xpcshell-based test </h3>
<p>Creating your first xpcshell-based test is easy. Create a file named <tt>test_first.js</tt> (unit test filenames must start <tt>test_</tt>) with the following:
</p>
<pre class="eval">function test_functionality_1()
{
  // do some complex action, return whether the action's results were what are expected here
  // this is just an example, so we return true
  return true;
}

function run_test()
{
  do_check_true(test_functionality_1());
}
</pre>
<p>This doesn't really test anything, but it gives you an idea how you'd actually write a test. If you want to execute the test, you must create a non-static build of the browser without <code>--disable-tests</code>. Add the test file to an existing set of tests (say, for example, {{ Source("tools/test-harness/xpcshell-simple/example/unit/") }}, but you'll want a different location when your test is posted for a review). You then execute your test using <code>make check</code>:
</p>
<pre class="eval">% /bin/sh
$ cd <b>objdir</b>
$ make -C tools/test-harness/xpcshell-simple/example check
...failure or success messages are printed to the console...
</pre>
<p>If you just want to run one test, directly, without any build system support, you can do this with something like the following commands. Suppose one wants to run "test_foo.js". </p>
<pre class="eval">$ /bin/sh
$ export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"
$ export TOPSRCDIR="c:/path/to/top/srcdir"
$ cd dist/bin
$ ./run-mozilla.sh ./xpcshell -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js \
    -f test_foo.js \
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js
...failure or success messages are printed to the console...
</pre>
<p>To add arbitrary (not necessarily xpcshell-based) tests and to get assistance with testing in general, follow the instructions at <a href="en/How_to_add_a_build-time_test">How to add a build-time test</a>.
</p>
<div class="note">{{ Fx_minversion_inline("3") }} <b>Note:</b> <p>You can now specify <code>check-one</code> instead of <code>check</code> on the command line to launch a single test ({{ Bug("404513") }}).
</p>
<pre class="eval">$ cd <b>objdir</b>
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-one
</pre>
</div>
<h3 name="Adding_to_your_test"> Adding to your test </h3>
<p>The test is executed by the test harness by calling the <code>run_test</code> function defined in the file. Anything you want to test must be spawned from this function.
</p><p>If you have a group of files in a directory and they all include some common code that needs to run before your tests, to set up resources for example, you can put one copy of that code in a source file whose name starts with <code>head_</code> and ends with <code>.js</code>. Any code that is common to your test and which needs to run after your tests can be put into a source file that starts with <code>tail_</code> and ends with <code>.js</code>. You should also name your test source files with names that start with <code>test_</code> and end with <code>.js</code>. Each of your test files needs to contain at least a "<code>run_test()</code>" function. If the "<code>run_test()</code>" function runs to completion without throwing an exception, the test succeeds. (If you need to do any asynchronous tests, see <code>do_test_pending</code> and <code>do_test_finished</code> below.)
</p><p>xpcshell tests have access to the following utility functions:
</p>
<dl><dt> <code>do_throw(<i>messageText</i>)</code>
</dt><dd> Call this function to report an error and exit the test. The argument is a string that will be reported in the test's log file.
</dd><dd> <i>Note</i>: <code>do_throw</code> can not be caught by a <code>try/catch</code> block, the test will always exit.
</dd></dl>
<dl><dt> <code>do_check_eq(<i>a</i>, <i>b</i>)</code>
</dt><dd> Call this function to assert that two objects are equal. If not equal, an exception is logged and the test case is halted.
</dd></dl>
<dl><dt> <code>do_check_neq(<i>a</i>, <i>b</i>)</code>
</dt><dd> Call this function to assert that two objects are not equal. If equal, an exception is logged and the test case is halted.
</dd></dl>
<dl><dt> <code>do_check_true(<i>expr</i>)</code>
</dt><dd> Call this function to assert that <code>expr</code> is equal to <code>true</code>.
</dd></dl>
<dl><dt> <code>do_check_false(<i>expr</i>)</code>
</dt><dd> Call this function to assert that <code>expr</code> is equal to <code>false</code>.
</dd></dl>
<dl><dt> <code>do_timeout(<i>delay</i>, <i>expr</i>)</code>
</dt><dd> Call this function to schedule a timeout. The given string expression will be evaluated after the specified delay (in milliseconds). Note that you must call <code>do_test_pending</code> so that the test isn't completed before your timer fires, and you must call <code>do_test_finished</code> when the actions you perform in the timeout complete, if you have no other functionality to test.)
</dd></dl>
<dl><dt> <code>do_test_pending()</code>
</dt><dd> Delay exit of the test until <code>do_test_finished()</code> is called. <code>do_test_pending()</code> may be called multiple times, and <code>do_test_finished()</code> must be paired with each before the unit test will exit.
</dd></dl>
<dl><dt> <code>do_test_finished()</code>
</dt><dd> Call this function to inform the test framework that an asynchronous operation has completed. If all asynchronous operations have completed (i.e. every <code>do_test_pending()</code> has been matched with a <code>do_test_finished()</code> in execution), then the unit test will exit.
</dd></dl>
<dl><dt> <code>do_import_script(<i>topsrcdirRelativePath</i>)</code>
</dt><dd> Imports the JavaScript file referenced by <code><i>topsrcdirRelativePath</i></code> into the global script context, executing the code inside it. The file specified is a file within the source tree. For example, to use it with the tree's <a href="en/HTTP_server_for_unit_tests">unit-testing HTTP server</a>, simply use <code>do_import_script("netwerk/test/httpserver/httpd.js")</code>. Currently this function is only used with the HTTP server, although it could be used for any script guaranteed to be pulled with the code for which the tests are written.
</dd></dl>
<dl><dt> <code>do_get_file(<i>topsrcdirRelativePath</i>)</code>
</dt><dd> Returns an <a href="en/NsILocalFile">nsILocalFile</a> object representing the given file (or directory) in the Mozilla tree, i.e. if <code><i>topsrcdirRelativePath</i></code> equals <code>netwerk/test/TestServ.js</code>, the returned file represents <code>mozilla/netwerk/test/TestServ.js</code>. The given path must be delimited with forward slashes; it is suggested that you end it with a forward slash if the file you are retrieving is a directory. You can use this to access test-specific auxiliary files if your test requires access to external files. Note also that you can use this to get directories; if you're accessing multiple files in the same location (e.g. <code>foo/bar/baz</code>, <code>foo/bar/quux</code>, and <code>foo/bar/eit</code>, it's probably worth getting the parent directory (in this case, <code>foo/bar/</code>) and cloning it to get each individual file.
</dd></dl>
<h3 name="Adding_your_tests_to_the_list_of_mozilla_tests"> Adding your tests to the list of mozilla tests </h3>
<p>You should store your tests near the source of code you want to test. For example, imagine you want to add tests for your component in <code>extensions/myextension/mycomponent</code>, you will add a <code>tests/</code> directory. You can store different kind of tests here (reftests etc.), and for xpcshell, you should create a <code>unit</code> directory and store all your javascript files inside it. Example: <code>extensions/myextension/mycomponent/tests/unit/test_foo.js</code>. The <code>unit</code> name is important because for the moment the xpcshell unit tests framework recognize only this name (especially when you want to launch a single test with check-interactive).
</p><p>Then you should create a <code>extensions/myextension/mycomponent/tests/Makefile.in</code> which will contain <code>XPCSHELL_TESTS = unit</code>. Of course, in the <code>extensions/myextension/mycomponent/Makefile.in</code> you will add <code>tests/</code> into the <code>DIRS</code> variable.
</p>
<h3 name="Running_unit_tests"> Running unit tests </h3>
<p>Tests should be run using <code>make -C <i>OBJDIR/path_to_tests</i> check</code>. Just <code>cd</code> to an appropriate directory (e.g. to run tests for a single module, go to <code><a href="en/MOZ_OBJDIR">MOZ_OBJDIR</a>/<i>path_to_tests</i>/test</code>) and run <code>make check</code>.
</p><p>To run a single test, use the <code>check-one</code> target, as described above.
</p>
<div class="note"><b>Note:</b> You must run <code>make -C <i>OBJDIR/path_to_tests</i></code> before trying to run tests; the tests are copied to a central location to be executed, and this copying occurs in the <code>libs</code> target of the build, which <code>make check</code> does not invoke. (this note seems to be deprecated for Firefox3 sources)</div>
<p>Log files end up in <code><a href="en/MOZ_OBJDIR">MOZ_OBJDIR</a>/_tests/xpcshell-simple/<i>module_name</i>/<i>tests_type</i>/*.log</code>. PASS/FAIL information for each test, as well as for the test suite as a whole, is printed to the console. It's also included in Tinderbox logs on tinderboxes which have <code>--enable-tests</code> specified, so if you break a test you can see any error messages it printed in the build log.
</p>
<h3 name="Running_unit_tests_under_a_C.2B.2B_debugger"> Running unit tests under a C++ debugger </h3>
<h4 name="Via_check-interactive"> Via <code>check-interactive</code> </h4>
<p>You can specify <code>check-interactive</code> when issuing the <code>make</code> command. This will cause your test to stop right before running so you can attach to it in a debugger (implemented in {{ Bug("382682") }}).
</p><p>Example:
</p>
<pre class="eval">$ cd <b>objdir</b>
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-interactive
# js&gt;_execute_test();
...failure or success messages are printed to the console...
# js&gt;quit();
</pre>
<p><i>Note: For comm-central builds (only?), the path passed to -C, giving the relative path from the base <b>source</b> directory to the test directory, must not contain the final /unit/ subdirectory, even if the actual SOLO_FILE is stored there!</i>
</p>
<pre class="eval">$ ls -la uriloader/exthandler/tests/unit/test_punycodeURIs.js
$ make SOLO_FILE="test_punycodeURIs.js" -C uriloader/exthandler/tests check-interactive
</pre>
<h4 name="Running_the_test_directly_under_gdb"> Running the test directly under gdb </h4>
<p>Getting your test script running in gdb is fairly simple. First, you have to export two variables like so:
</p>
<pre class="eval"><code>export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"</code>
<code>export TOPSRCDIR="c:/path/to/top/srcdir"</code>
</pre>
<p>Then, run <code>OBJDIR/dist/bin/run_mozilla.sh -g xpcshell</code>, set your breakpoints or whatever else you need to do, then type <code>run -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js -f path/to/test_file.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js</code>. If you have head files that are automatically included, you should also include those before you include your test file with the -f option.
</p>
Revert to this revision