mozilla

Gaia UI Tests: Running Tests

This article provides a detailed rundown of how to run the actual gaia-ui-tests suite that resides inside Gaia itself, on B2G Desktop, and real Firefox OS devices. The Gaia project uses Treeherder to do per-commit continuous integration and pull request tests.

The gaia-ui-tests have been developed with the capability to run on a real device device, emulators, and the B2G Desktop client. By far the easiest method is to run against B2G Desktop. Please note that you should choose the right branch of Gaia to run against the version of Firefox OS that you are using.

Testing on B2G Desktop

The B2G desktop client is as it sounds: it's a desktop version of B2G that you can use to perform web app tests on Firefox OS from the comfort of your desktop. If you are not familiar with it, you can find more out by reading Using the B2G desktop client. This section will show you how to run gaia-ui-tests on B2G desktop.

Prerequisites

  • A B2G desktop build. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • The Gaia Github repository cloned and checked out. You will also need to generate a profile. Note that you must not pass DEBUG=1 while generating the profile - using a debug gaia profile will cause gaia-ui-tests to fail to connect with the b2g desktop instance.
  • A testvars.json file; see Configuring test variables.

The desktop B2G client is ideal for developing tests when a real device is unavailable and there is no need for phone-level functions. It is fast, accessible, and will run on Linux, Mac, and Windows!

Running the tests

We'll look at running new tests; we'll install the in-tree version of gaiatest directly from the Gaia repository so that our changes are picked up immediately. In your cloned version of Gaia, navigate to gaia/tests/python/gaia-ui-tests and run the setup command:

python setup.py develop

This might give permission denied errors when not running it inside a virtual environment. In that case, install virtualenv and virtualenvwrapper, if you haven't already done so:

pip install virtualenv
pip install virtualenvwrapper (perhaps need to use sudo here)
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bash (rc or _profile)

Then, create and enter the virtual environment:

mkvirtualenv gaia-ui-test

Via Marionette, gaiatest is able to launch the B2G Desktop when we start the test. To do this, you need to pass in the path to the B2G binary file and B2G profile. In the below example we have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped B2G Desktop to after you downloaded it. The command to run all the tests is:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart --type=b2g ./gaiatest/tests/functional/manifest.ini

Tests that are incompatible with the B2G Desktop client — for example those that make calls and send SMS — are automatically skipped from the test run.

You can run a test directly by calling the file on the command line:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py

Testing on Firefox OS devices

Testing on real Firefox OS devices is more complicated, but for obvious reasons gives the most accurate results. With access to all available device APIs, hardware features, and so forth. However, you can test on the emulator if you don't have a real Firefox OS device, or if the features you're using don't require real hardware. You can also test general functionality in the emulator.

Prerequisites

Running the tests

The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the gaia/tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

python setup.py develop

Note: Before you start to run the test, read through this warning in order to acknowledge that running gaia-ui-tests on a device may cause data to be deleted from the device!

Debug Firefox OS builds: Firefox OS builds built with DEBUG=1 will need a greater timeout than the default (20 seconds). Try adding to the command line --timeout 60000 or higher, depending upon the speed of the device.

Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb. Run the following command:

adb forward tcp:2828 tcp:2828

Please be reminded that you'll need to start your emulator before the command, if you are not running on a real Firefox OS device.

You can use the following command to execute all the gaia-ui-tests:

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

Or use the following if you want to run a single test (substituting the last part for the actual test you want to run):

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/tests/functional/clock/test_clock_create_new_alarm.py

After the test, you will see all the pass/fail results and stacktraces for each file.

Testing locally with Treeherder configuration for Gecko and Gaia developers

In this section we will focus upon replicating a Treeherder test locally. Before testing locally with a Treeherder configuration, you need to be aware that it builds a Gaia profile separately. The profile that comes packaged with the Treeherder build or rel-eng build is NOT the same Gaia profile that Treeherder uses.

Prerequisites

  • A B2G desktop build. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • The Gaia Github repository checked out (for making the Gaia profile, and installing gaiatest and the test files).
  • A testvars.json file; see Configuring test variables.

Building a Gaia profile with Treeherder's  settings

Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '0',
            'NOFTU': '0'}

These settings can drastically change the way desktop B2G behaves. It is very important to match Treeherder's settings.

The next step is to checkout the git commit that matches the hg commit that Treeherder ran with. You can also find the hg commit in the "brief log" of the Treeherder run but you will need to correlate that with the git commit yourself.

The gaia test repository includes a make script for building a Gaia profile. Navigate to the Gaia repo that you have cloned locally, and build the profile using the environment variables we found on Treeherder:

DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=0 NOFTU=0 make

Running the tests

Using the same git commit that Treeherder used to run the test, go to the gaia/tests/python/gaia-ui-tests directory and install gaiatest using the following command:

python setup.py develop

Now that you have built the Gaia profile, the only significant difference to running these tests is that you must direct gaiatest to the binary location and profile location.

Inside the gaia-ui-tests directory, run the below command, with the following substitutions made:

  • $BINARY_PATH: The path to the b2g-bin file in the directory you have built or unpacked desktop B2G to.
  • $PROFILE_PATH: The path to your locally build Gaia profile. If built from the Gaia repository it will be ~/gaia/profile.
gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/manifest.ini --type=b2g

Note: There is no need to forward port 2828 unless you have set the Marionette server to use a different default port.

Important: Try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.

Finding the HTML report for a completed Treeherder job

All Treeherder jobs (both pass and fail) have an HTML report output generated and stored.

  1. For the commit and Gip job, click the chunk of interest (e.g., f1, f2).
  2. In the status bar at the bottom of the screen find the section titled: output.html.
  3. Click link to view the report.

Testing against an already running Desktop B2G binary

In this section we will look at running the tests against an already running Desktop B2G binary with modified source code.

Prerequisites

  • A B2G desktop build with your Gecko changes. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • Your debugger of choice attached to the B2G binary.
  • The Gaia Github repository checked out (for installing gaiatest and the test files).
  • A testvars.json file; see Configuring test variables.

Running the tests

Running tests against an existing b2g process is quite easy — you've already gone through the hard parts to build B2G!

If you have not already installed gaiatest then do so now by going to gaia/tests/python/gaia-ui-tests and running the following:

python setup.py develop

Once you have started the b2g binary process and attached your debugger you need to direct gaiatest to the port using the --address command. Do not use --binary or --profile, otherwise gaiatest will just send commands directly to the port without attempting to start or close the binary.

gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py

Note: We have omitted the --restart command too. As gaiatest does not attempt to stop or start the binary, some data from your test run may be left behind in Firefox OS databases or even in the DOM. It is your responsibility to reset B2G back to a basic state before trying to run the test for a second time.

Configuring test variables

You can customize variables used during the test using a testvars.json file. The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data. Please read the warning before you set up your test variables file.

We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

When running your tests add the argument: --testvars=(filename).json

How to configure WiFi using testvars.json (for devices)

By setting the WiFi key in testvars.json you can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

No WiFi authentication:

"wifi": { "ssid": "MyNetwork"} 

WEP authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

WPA-PSK authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

Note: Due to bug 775499, WiFi connections via WPA-EAP are not possible at this time.

Troubleshooting

This section details some common error messages, offering potential solutions for each.

Common problems before the test has started

Error Message Corresponding Actions
error: [Errno 111] Connection refused

Reissue the "adb forward tcp:2828 tcp:2828" command

B2G process is not running

Element ... not visible before timeout Make sure the element would display on the app you test
TimeoutException: Condition timed out make sure the condition on the app is the same as your expectations
marionette.errors.MarionetteException: localhost:2828 is unavailable. Something is blocking the port, often the `adb forward` command. `adb kill-server` will resolve that, or closing the app that is keeping the port open
ImportError: No module named bluetooth

Replace --type=b2g in your command with --type=b2g-bluetooth
OR
Install pybluez Python bluetooth package

OSError: [Errno 2] No such file or directory: '~/moz/gaia/profile' Check to make sure that you have not used ~ on the command line where you meant $HOME.

 

Document Tags and Contributors

Last updated by: Sheppy,
Hide Sidebar