Writing xpcshell-based unit tests

From MDC

The xpcshell tool can be used to test certain kinds of functionality. Anything available to the XPCOM layer (through scriptable interfaces) can be tested with xpcshell. See Mozilla automated testing and category for pointers to more information.

[edit] Your first xpcshell-based test

Creating your first xpcshell-based test is easy. Create a file named test_first.js (unit test filenames must start test_) with the following:

function test_functionality_1()
{
  // do some complex action, return whether the action's results were what are expected here
  // this is just an example, so we return true
  return true;
}

function run_test()
{
  do_check_true(test_functionality_1());
}

This doesn't really test anything, but it gives you an idea how you'd actually write a test. If you want to execute the test, you must create a non-static build of the browser without --disable-tests. Add the test file to an existing set of tests (say, for example, tools/test-harness/xpcshell-simple/example/unit/, but you'll want a different location when your test is posted for a review). You then execute your test using make check:

% /bin/sh
$ cd objdir
$ make -C tools/test-harness/xpcshell-simple/example check
...failure or success messages are printed to the console...

If you just want to run one test, directly, without any build system support, you can do this with something like the following commands. Suppose one wants to run "test_foo.js".

$ /bin/sh
$ export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"
$ export TOPSRCDIR="c:/path/to/top/srcdir"
$ cd dist/bin
$ ./run-mozilla.sh ./xpcshell -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js \
    -f test_foo.js \
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js
    -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js
...failure or success messages are printed to the console...

To add arbitrary (not necessarily xpcshell-based) tests and to get assistance with testing in general, follow the instructions at How to add a build-time test.

$ cd objdir
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-one

[edit] Adding to your test

The test is executed by the test harness by calling the run_test function defined in the file. Anything you want to test must be spawned from this function.

If you have a group of files in a directory and they all include some common code that needs to run before your tests, to set up resources for example, you can put one copy of that code in a source file whose name starts with head_ and ends with .js. Any code that is common to your test and which needs to run after your tests can be put into a source file that starts with tail_ and ends with .js. You should also name your test source files with names that start with test_ and end with .js. Each of your test files needs to contain at least a "run_test()" function. If the "run_test()" function runs to completion without throwing an exception, the test succeeds. (If you need to do any asynchronous tests, see do_test_pending and do_test_finished below.)

xpcshell tests have access to the following utility functions:

do_throw(messageText)

Call this function to report an error and exit the test. The argument is a string that will be reported in the test's log file.

Note: do_throw can not be caught by a try/catch block, the test will always exit.

do_check_eq(a, b)

Call this function to assert that two objects are equal. If not equal, an exception is logged and the test case is halted.

do_check_neq(a, b)

Call this function to assert that two objects are not equal. If equal, an exception is logged and the test case is halted.

do_check_true(expr)

Call this function to assert that expr is equal to true.

do_check_false(expr)

Call this function to assert that expr is equal to false.

do_timeout(delay, expr)

Call this function to schedule a timeout. The given string expression will be evaluated after the specified delay (in milliseconds). Note that you must call do_test_pending so that the test isn't completed before your timer fires, and you must call do_test_finished when the actions you perform in the timeout complete, if you have no other functionality to test.)

do_test_pending()

Delay exit of the test until do_test_finished() is called. do_test_pending() may be called multiple times, and do_test_finished() must be paired with each before the unit test will exit.

do_test_finished()

Call this function to inform the test framework that an asynchronous operation has completed. If all asynchronous operations have completed (i.e. every do_test_pending() has been matched with a do_test_finished() in execution), then the unit test will exit.

do_import_script(topsrcdirRelativePath)

Imports the JavaScript file referenced by topsrcdirRelativePath into the global script context, executing the code inside it. The file specified is a file within the source tree. For example, to use it with the tree's unit-testing HTTP server, simply use do_import_script("netwerk/test/httpserver/httpd.js"). Currently this function is only used with the HTTP server, although it could be used for any script guaranteed to be pulled with the code for which the tests are written.

do_get_file(topsrcdirRelativePath)

Returns an nsILocalFile object representing the given file (or directory) in the Mozilla tree, i.e. if topsrcdirRelativePath equals netwerk/test/TestServ.js, the returned file represents mozilla/netwerk/test/TestServ.js. The given path must be delimited with forward slashes; it is suggested that you end it with a forward slash if the file you are retrieving is a directory. You can use this to access test-specific auxiliary files if your test requires access to external files. Note also that you can use this to get directories; if you're accessing multiple files in the same location (e.g. foo/bar/baz, foo/bar/quux, and foo/bar/eit, it's probably worth getting the parent directory (in this case, foo/bar/) and cloning it to get each individual file.

[edit] Adding your tests to the list of mozilla tests

You should store your tests near the source of code you want to test. For example, imagine you want to add tests for your component in extensions/myextension/mycomponent, you will add a tests/ directory. You can store different kind of tests here (reftests etc.), and for xpcshell, you should create a unit directory and store all your javascript files inside it. Example: extensions/myextension/mycomponent/tests/unit/test_foo.js. The unit name is important because for the moment the xpcshell unit tests framework recognize only this name (especially when you want to launch a single test with check-interactive).

Then you should create a extensions/myextension/mycomponent/tests/Makefile.in which will contain XPCSHELL_TESTS = unit. Of course, in the extensions/myextension/mycomponent/Makefile.in you will add tests/ into the DIRS variable.

[edit] Running unit tests

Tests should be run using make -C OBJDIR/path_to_tests check. Just cd to an appropriate directory (e.g. to run tests for a single module, go to MOZ_OBJDIR/path_to_tests/test) and run make check.

Log files end up in MOZ_OBJDIR/_tests/xpcshell-simple/module_name/tests_type/*.log. PASS/FAIL information for each test, as well as for the test suite as a whole, is printed to the console. It's also included in Tinderbox logs on tinderboxes which have --enable-tests specified, so if you break a test you can see any error messages it printed in the build log.

[edit] Debugging unit tests in gdb

Getting your test script running in gdb is fairly simple. First, you have to export two variables like so:

export NATIVE_TOPSRCDIR="c:\path\to\top\srcdir"
export TOPSRCDIR="c:/path/to/top/srcdir"

Then, run OBJDIR/dist/bin/run_mozilla.sh -g xpcshell, set your breakpoints or whatever else you need to do, then type run -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/head.js -f path/to/test_file.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/tail.js -f path/to/top/srcdir/tools/test-harness/xpcshell-simple/execute_test.js. If you have head files that are automatically included, you should also include those before you include your test file with the -f option.

$ cd objdir
$ make SOLO_FILE="test_foo.js" -C tools/test-harness/xpcshell-simple/example check-interactive
# js>_execute_test();
...failure or success messages are printed to the console...
# js>quit();