Firefox OS mochitests

Mochitest is an automated testing framework built on top of the MochiKit JavaScript libraries, which provides functional and API tests. It's just one of the automated regression testing frameworks used by Mozilla. Tests report success or failure to the test harness using JavaScript function calls. This article takes you through the steps needed to run Mochitests on Firefox OS, via both B2G Desktop and device/emulator builds.

Running mochitests on B2G desktop builds

The first case we'll cover is running B2G on B2G desktop. You'll need to install Gaia and build a profile before you can get started.

Using mach

Mach is a command-line interface to help developers perform common tasks, let's look at how to use it run mochitests.If you have a local B2G Desktop build with marionette enabled (specify ENABLE_MARIONETTE=1 in your mozconfig), you can use mach to run all of the mochitests. Run the following command from the root of your B2G source tree:

./mach mochitest-b2g-desktop --profile path/to/non/debug/gaia/profile

Alternatively you can set the $GAIA_PROFILE environment variable to avoid having to pass it in all the time.

To run a specific test, you can run the following:

export GAIA_PROFILE=path/to/non/debug/gaia/profile
./mach mochitest-b2g-desktop test_path

To see mochitest options:

./mach help mochitest-b2g-desktop

Note: Some contributors have been working on a project called b2g-commands, which aims to bring some of the commands available to desktop Firefox builds to B2G, such as easier diagnostics and build environment bootstrapping. Find more out by reading Add more Mach to your B2G.

Using the mochitest runner

You can run mochitest-plain on B2G Desktop builds.

Pre-requisites

  • You need to be using a Marionette-enabled build (compiled with ENABLE_MARIONETTE=1 in mozconfig.)
  • You need to have manually generated a gaia profile (which you should already have if you've been running a B2G desktop build manually); this must be a non-DEBUG profile
  • You cannot use a Gaia profile that was made with a custom GAIA_DOMAIN.

Setup for mozilla-b2g18 trees

cd $GECKO_OBJ_DIR
python $GECKO_SRC_DIR/python/virtualenv/virtualenv.py venv
source venv/bin/activate
cd $GECKO_SRC_DIR/testing/marionette/client
python setup.py develop

Note: you may need to install setuputils.py before you run setup.py

Setup for trunk trees, including mozilla-central

cd $GECKO_OBJ_DIR
source _virtualenv/bin/activate

Running the tests

You can then run mochitests using the following commands:

cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --desktop --console-level INFO --profile /path/to/gaia/profile --app /path/to/b2g

The test runner will launch your b2g desktop build and run the tests; all B2G mochitests by default, which is very time consuming. Other common options are as follows:

  • To run specific tests, pass in --test-path.
  • You can optionally add --total-chunks and --this-chunks arguments as you would with regular desktop mochitest.
  • Finally, you can use python runtestsb2g.py --help for a full list of supported arguments.

Running mochitests on device and emulator builds

You can run mochitest-plain on B2G device builds. Currently, these work best when run on an emulator.

Using mach

If you have a local emulator build that is made against a trunk Gecko branch (not mozilla-b2g18), the easiest way to run mochitests on it is with mach.  To run all of the tests, do the following:

cd $B2G_DIR
./mach mochitest-remote

To run a specific test, use this:

./mach mochitest-remote test_path

Note: Test paths are relative to gecko's source directory, e.g. content/base/test/test_CrossSiteXHR.html

To see mochitest options, you cna use this:

./mach help mochitest-remote

Using the mochitest runner

Similar to before, you can also run mochitest on device or emulator builds of B2G using the mochitest runner. The prerequisites are a little more complex.

Pre-requisites

  • You need to build B2G for the target you're testing (see Building and installing Boot to Gecko).
  • You cannot use a Gaia build that was made with a custom GAIA_DOMAIN.
  • You need to have a desktop version of xpcshell (it needs to run on the host); you will have this already if you have a copy of Firefox on your system.  Alternatively, you can download a copy for Linux from http://people.mozilla.com/~ahalberstadt/getb2g/xre.zip.
  • You need to install some Python packages that are needed, either in a virtualenv or otherwise. Read on for more details.

Setup for mozilla-b2g18 trees

cd $GECKO_OBJ_DIR
python $GECKO_SRC_DIR/python/virtualenv/virtualenv.py venv
source venv/bin/activate
cd $GECKO_SRC_DIR/testing/marionette/client
python setup.py develop

Note: you may need to install setuputils.py before you run setup.py

Setup for trunk trees, including mozilla-central

cd $GECKO_OBJ_DIR
source _virtualenv/bin/activate

Running the tests

You can then run mochitest-plain using one of the following sets of terminal commands.

To run tests on a device, both the device and the host computer must be on the same wireless network, since the device will request files from an HTTP server that's started by mochitest on the host computer.  To start the tests, use the following:

adb forward tcp:2828 tcp:2828
cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --b2gpath $B2G_HOME --xre-path /path/to/dir/containing/desktop/xpcshell --console-level INFO \
  --httpd-path ./

If you're using an emulator, use the following:

cd $GECKO_OBJ_DIR/_tests/testing/mochitest
python runtestsb2g.py --b2gpath $B2G_HOME --xre-path /path/to/dir/containing/desktop/xpcshell --console-level INFO \
  --httpd-path ./ --emulator arm

Warning: Running mochitests on a B2G emulator on OS X is currently not supported; use Linux instead.

The above commands will run all B2G mochitests by default, which is very time consuming. Other common options are as follows:

  • To run specific tests, pass in --test-path.
  • You can optionally add --total-chunks and --this-chunks arguments as you would with regular desktop mochitest.
  • Finally, you can use python runtestsb2g.py --help for a full list of supported arguments.

After you invoke runtestsb2g.py, the test runner will launch the emulator for you (if you're running the tests on an emulator) or reboot your device (if you're running the tests on a device), and start running the tests. Because the emulator is slow, and it is necessary to push a test profile to the emulator and restart B2G, the tests can take a few minutes to start. Before they start, you will just see a black or white screen. After they start, you should see the test log being dumped to the console.

When the tests are done, the emulator is shut down for you, or if you're using a device, the device is rebooted.

Note: There is a current bug 780034 which prevents mochitest from terminating cleanly if you just run a single test, as opposed to a set of tests. In this case, when you notice the test has ended, you can press CTRL+C to terminate the testrunner.

Running mochitest with a downloaded emulator

If you've built B2G for another config (like otoro) and want to run the tests on an emulator, you can do so without building an emulator yourself. Just download the latest trunk arm emulator, and use the same instructions as above, replacing the --b2gpath $B2G_HOME argument for runtestsb2g.py with --b2gpath /path/to/unpacked/emulator.

Running mochitest with a downloaded emulator and downloaded tests

You can also run mochitests on an emulator without building or cloning anything.  To do this, you need to download the latest trunk arm emulator and unpack it, and download the latest tests.zip file (from the same place as the emulator) and unpack it. 

You'll also need virtualenv installed on your system.

Then you can set up your environment and run the tests:

virtualenv venv
source venv/bin/activate
cd $TESTS_DIR/marionette
python setup.py develop
cd $TESTS_DIR/mochitests 
python runtestsb2g.py --b2gpath /path/to/extracted/emulator --xre-path /path/to/desktop/xpcshell --console-level INFO \
  --httpd-path ./ --emulator arm

Technical Implementation Details

There are a few things that happen when you run B2G mochitests.

  1. The homescreen system app is replaced by a special certified app called test-container (note this is checked in to gaia). In theory this test-container is meant to be generic such that any test harness can use it, but so far only mochitests do.
  2. The test-container app exposes a single <iframe mozbrowser mozapp> in which the mochitests are loaded (mochitests have their own manifest.webapp).
  3. The automation replaces the homescreen app with the test-container app using a pref.
  4. The automation loads the mochitest app into the iframe the test-container app exposes (this script is executed with chrome privileges by marionette).
  5. The automation loads special powers into the child frame that is now running mochitests and sets up the message manager (this lets mochitests access some privileged apis).
  6. The automation loads the mochitest url into the mochitest app's iframe. This kicks off the tests.

Note: If you're interested in why things work this way, the short answer is to satisfy permissions testing requirements. For a more detailed answer see bug 798580 and bug 814140.

 

Document Tags and Contributors

Last updated by: billmccloskey,
Hide Sidebar