The teams behind our automation harnesses have been working hard to expand our automated testing infrastructure to accomodate Firefox OS as well as writing new harnesses to specifically target testing a phone OS rather than a browser (which our old harnesses were specialized for). Due to the architecture of Firefox OS all these test harnesses will work and will apply, but that also makes things rather complicated if you are a developer and just want to run some tests to see if your patch worked or not. This article aims to make sense of all the available testing resources at Mozilla.
If you are a Gaia or App developer, or if you're a Gecko developer interested in doing a deeper "end user" style test where you actually exercise the OS, then you'll need to look into the Gaia test suites. There are two primary test suites:
- Gaia UI Tests: These tests are Python-based, and can be run on both desktop and devices. These tests test various aspect of the Gaia UI such as functional tests (e.g. finding stations on the FM radio app, capturing a photo using the Camera app) and accessibility tests (e.g. enabling and disabiling the screenreader.)
Which one you choose really depends on your preferred toolchain, and what things you want to test.
Let's now move and look at running these tests.
Running the Gaia UI Tests
The Gaia UI Test suite can be run on real devices and B2G Desktop builds, but in this section we'll concentrate on running them on a real device, as real devices are always best where possible.
Note that this test is destructive and as such, you should back up anything you care about on the phone before running these tests. Depending on which tests you run, they can also make phone calls. So be aware that you want to be very careful about what you run and how you back up the phone, remove the SIM card, etc. That said, if you've already created an engineering build they are really easy to get running. Here's how.
One Time Set up
You only need to perform the following steps once, assuming you do not change the location of your Gaia directory. Create a python virtualenv (install the virtualenv tool first if you haven't already), activate it, and install the gaia UI test tool into your virtualenv. By creating the virtual environment using the steps below, you ensure that you are running the gaia UI test harness code that lives in your Gaia repo (that's useful in case you need to debug anything).
$ virtualenv gaia_ui_venv # This will create a gaia_ui_venv directory where the virtual environment lives. It can be anywhere on your system. $ source gaia_ui_venv/bin/activate # This activates our virtualenv (gaia_ui_venv)$ cd <b2groot>/gaia/tests/python/gaia-ui-tests;python setup.py develop # This installs the gaia ui harness into your virtual environment.
If you have already created a virtual environment for gaia ui tests, you can simply do the following:
$ source gaia_ui_venv/bin/activate
To Run the Tests
First you need to create the testvars file. To do this, copy the standard one over, and add in the attributes to turn off the warnings that this test will destroy all content on your phone. These are good tests, they leave no state around and as such, you will need to make sure your phone is backed up before running them. Instructions from here on will assume you've activated the virtual environment and are working in the
(gaia_ui_venv)$ cp gaiatest/testvars_template.json testvars.json # Now edit your copy of testvars.json and add in the following attributes into the json: "acknowledged_risks": true, "skip_warning": true,
Now you just need to connect our phone via USB, forward the marionette port so your test runner can access it and run our tests. The tests are in
gaiatest/tests and you can pick whichever one you want to run. For example, if you wanted to run contacts tests you would do the following:
(gaia_ui_venv)$ adb forward tcp:2828 tcp:2828 (gaia_ui_venv)$ gaiatest --testvars=testvars.json --address=localhost:2828 gaiatest/tests/functional/contacts/
Note: If you are using the Aries (Sony Xperia Z3C), you'll need to enable ADB root with the command
Note: To find out what UI tests are available, browse through the gaiatest directories inside the Gaia repo.
To get out of the python virtualenv, just use the special virtualenv command
(gaia_ui_venv)$ deactivate $
Note: To learn more about the Gaia UI Tests and find more detailed information, move on to the Gaia UI Tests pages.
Running the Gaia Integration tests
To run the Gaia Integration tests tests you currently have to use a B2G Desktop build (note that these are also going to be available for devices soon as well). Let's look at how this is done.
These just require a Gaia tree and NodeJS to be installed on your computer; the following command will do the rest:
$ cd gaia $ make test-integration
That's it — this instruction will download a B2G desktop build, and start running the tests in that build.
Note: To find out what integration tests are available, look in the apps directory in the Gaia repo; integration tests can be found in
As always, work is underway to make all our tests easier to run both locally for developers as well as in our automation systems. Feel free to drop into the #ateam channel any time you have questions about test automation for Firefox OS or any of the Mozilla automation tools.