Gaia unit tests

This article is in need of a technical review.

As part of the Gaia/B2G source code, we have made various unit tests available to run, for testing different aspects of Gaia and B2G. This article explains how to access these.

Note: This document assumes you fully understand how to work within Gaia and B2G. See Hacking Gaia to get started with Gaia.

Running unit tests

You can run the unit tests on B2G desktop or Firefox Nightly. You also need the latest Gaia repo. For most of the functionality, you must also have Node.js and NPM installed.

Note: If the following commands fail with cryptic errors while installing test-agent dependencies, it might be due to your Node.js/NPM versions being too old. Read Installing Node.js via package manager in order to install the very latest versions and delete the /tools/test-agent/node_modules folder.)

In the Gaia repo comes the handy bin/gaia-test script, that helps running the tests in an easy way.

Be careful: this script will generate a profile suited for unit tests in profile-debug. If you already have another profile in this directory it will be overwritten. Bug 980496 aims to make this configurable.

Launching the test runner in Firefox

This will run the test server and launch your default Firefox as found in the path:

bin/gaia-test

Choosing Firefox binary

You can export the FIREFOX environment variable to your firefox binary. For example, on MacOS X:

export FIREFOX=/Applications/FirefoxNightly.app/Contents/MacOS/firefox

Alternatively, you can pass it as argument to bin/gaia-test:

bin/gaia-test <gaia directory> <firefox path>

Launching the test runner in B2G Desktop

This will download and launch B2G Desktop:

bin/gaia-test -d

Run the tests from the Web Interface

You can simply click on specific tests and then the Execute button.

Run the tests from the command line

With the WebSocket server running, and the Test Agent app running in B2G Desktop/Firefox Nightly, run the following command:

make test-agent-test

If you only want to run one app's tests you can specify which via the APP env variable:

make test-agent-test APP=calendar

You can also optionally provide a reporter to format the test output:

make REPORTER=List test-agent-test
Note: Not all reporters work, since we currently do not support Doc.

Run the tests as you save

When the server is running, the tests for a file are run automatically when a file is saved or even just touched:

  • When you save a test file, the test file itself is run
  • When you save another file, it finds a matching file in the test/unit directory, by suffixing the file name with _test.js.

Note: It watches only existing files so if you create a new file, you have to restart the agent.

Running tests like TBPL does

Gaia unit tests in TBPL are run using a separate runner; this explains how to use it.  Please consult the virtualenv docs if you're not familiar with using a Python virtualenv.

virtualenv venv
source venv/bin/activate
cd $GAIA/tests/python/gaia-unit-tests
python setup.py develop
cd gaia_unit_test
python main.py --binary /path/to/b2g/desktop/build --profile /path/to/gaia/profile

Note: When specifying the path to the B2G desktop build, you should specify the path to b2g-bin, if it exists, otherwise use b2g.

The Gaia profile must be made using the following:

NO_LOCK_SCREEN=1 DEBUG=1 DESKTOP=0 make

By default, this profile will be generated in $GAIA/profile-debug. bin/gaia-test generates the same profile so you don't need to regenerate it if you already run gaia-test.

Disabling a gaia unit test in TBPL

TBPL uses a blacklist to exclude certain gaia unit tests from being run.  To prevent a test from running in TBPL, add its path to https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-unit-tests/gaia_unit_test/disabled.json.

Setting up your Gaia app

Although this guide should help make things easier, the best way to learn how to write, set up, and run tests is currently still to look at the source code; in particular, take a look at the gallery tests.

Using mocks

TBD

Advanced: what does the script do?

Generating a profile

You need a profile that is generated by this command:

NO_LOCK_SCREEN=1 DEBUG=1 DESKTOP=0 make

This generates a debug profile in gaia/profile-debug, overriding a previous profile if you already have one.

  • DEBUG=1 enables the httpd.js extension that makes it possible to directly use the files from the apps/ directory.
  • NO_LOCK_SCREEN=1 disables the lock screen, which is necessary in B2G Desktop because it's not possible to unlock it using the mouse.
  • DESKTOP=0 disables the other addons we normally use in DEBUG mode to run Gaia in Firefox.

Launch the WebSocket server

Test agent (the test runner) ships with a built in WebSocket server that lets you remotely message the browser or device to queue a test run. Often you will want to develop with time saving features like a file watcher that will run your tests when a test file or implementation changes. To take advantage of these features you need to start the server:

make test-agent-server

Using the WebSocket server provides other tools such as a command line reporter for test results (watch the terminal you ran the command from), a Growl reporter, syntax error notifications, and more.

The agent also watches for modifications in files, and automatically runs the associated tests. It runs when you save the test or if you save the tested file (we use the convention where the test filename is the tested filename with _test appended, see below for more examples). It watches only existing files so if you create a new file, you have to restart the agent.

Running unit tests with Firefox Nightly

You can launch Gaia in Firefox Nightly with the following commands:

cd <path to gaia>
<path to nightly>/firefox --no-remote -profile <path to gaia>/profile-debug/ http://test-agent.gaiamobile.org:8080/

Note: In Mac OSX, The profile path should be absolute path

You can use Firebug or the integrated debugger to debug the tests; use the debugger keyword to break in the debugger.

Running unit tests with B2G Desktop

Launch Gaia and start the "Test Agent" app. From the Test gent app you can select tests to run from the UI.

Advanced: how does the test-agent work?

The Test Agent lives in its own Github repository. You can have look there to understand how it works under the hood.

Hide Sidebar