Gaia UI Tests: Running Tests

This article provides a detailed rundown of how to run the actual gaia-ui-tests suite that resides inside Gaia itself, on B2G Desktop, and real Firefox OS devices. The Gaia project uses Travis CI to do per-commit continuous integration and pull request tests.

Note: You can browse a list of over 32000 builds (and counting) that have been run via Travis. Besides linting, unit-tests, and marionette js tests, Gaia also runs buildscript tests and gaia_ui_tests (based on marionette's Python version) in Travis. The details of running scripts are defined in tests/travis_ci. If you see the gaia_ui_tests error on Travis and want to reproduce it locally, you could refer to tests/travis_ci/gaia_ui_tests/.

The gaia-ui-tests have been developed with the capability to run on a real device device, emulators, and the B2G Desktop client. By far the easiest method is to run against B2G Desktop. Please note that you should choose the right branch of Gaia to run against the version of Firefox OS that you are using.

Testing on B2G Desktop

The B2G desktop client is as it sounds: it's a desktop version of B2G that you can use to perform web app tests on Firefox OS from the comfort of your desktop. If you are not familiar with it, you can find more out by reading Using the B2G desktop client. This section will show you how to run gaia-ui-tests on B2G desktop.

Prequisites

  • A B2G desktop build.  You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • The Gaia Github repository cloned and checked out.
  • A testvars.json file created (as expained in the relevant section below.)

The desktop B2G client is ideal for developing tests when a real device is unavailable and there is no need for phone-level functions. It is fast, accessible and will run on Linux, Mac and Windows!

Running the tests

We'll look at running new tests; we'll install the in-tree version of gaiatest directly from the Gaia repository so that our changes are picked up immediately. In your cloned version of Gaia, navigate to gaia/tests/python/gaia-ui-tests and run the setup command:

python setup.py develop

Via Marionette, gaiatest is able to launch the B2G Desktop when we start the test. To do this you need to pass in the path to the B2G binary file and B2G profile. In the below example we have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped B2G Desktop to after you downloaded it. The command to run all the tests is:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart --type=b2g ./gaiatest/tests/functional/manifest.ini

Tests that are incompatible with the B2G Desktop client — for example those that make calls and send SMS — are automatically skipped from the test run.

You can run a test directly by calling the file on the command line:

gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py

Testing on Firefox OS Devices

Testing on real Firefox OS devices is more complicated, but it gives by far the most accurate results, with access to all the real device APIs, etc.

Prerequisites

Running the tests

The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the gaia/tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

python setup.py develop

Note: Before you start to run the test, read through this warning in order to acknowledge that running gaia-ui-tests on a device may cause data to be deleted from the device!

Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb. Run the following command:

adb forward tcp:2828 tcp:2828

You can use the following command to execute all the gaia-ui-tests:

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

Or use the following if you want to run a single test (substituting the last part for the actual test you want to run):

gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/tests/functional/clock/test_clock_create_new_alarm.py

After the test, you will see all the pass/fail results and stacktraces for each file.

Note: You may get an error when running these tests — ImportError: No module named bluetooth. If this happens, replace the --type=b2g part of the command you're running with --type=b2g-bluetooth to exclude the bluetooth tests. Alternatively you can install PyBluez python package which will allow your host runner to work with the device via bluetooth.

Running tests using Travis for Gecko and Gaia developers

Travis is a continuous integration (CI) platform that we use to automate running of gaia-ui-tests (and other things.) In this section we will explore how you can use this to your advantage if you are intending to run a large number of tests.

Running tests using our Travis configuration is quite easy — the install script will download Gecko and build a Gaia profile for you. The Travis install script uses mozilla-download which in turn uses nodejs and npm to download the latest Gecko version.

Prerequisites

  • The Gaia Github repository checked out (for making the Gaia profile, and installing gaiatest and the test files).
  • A testvars.json file created (as expained in the relevant section below.)
  • nodejs and npm installed (these are used to download Gecko.)

Installing Gecko and building Gaia using our Travis script

Note: If you already have a b2g/ directory in Gaia then the Travis install script will skip the download step. If you need the latest Gecko then delete the b2g/ directory before running the script.

The travis scripts must be run from the gaia/ root directory. Run the following:

tests/travis_ci/gaia_ui_tests/install

This install script will create a Python virtual environment (package sandbox) and installs the packages required for gaia-ui-tests to run, including a new B2G Desktop version and a desktop Gaia build. The virtual environment is installed in the travis_venv directory.

Running the tests

Running the tests is a matter of executing the script file, which wraps around the default command line:

tests/travis_ci/gaia_ui_tests/script

This runs the following:

$ export GAIATEST_ACKNOWLEDGED_RISKS=true
$ export GAIATEST_SKIP_WARNING=true
$ root=tests/python/gaia-ui-tests/gaiatest
$ b2g=`find b2g -follow -name "b2g-bin" | tail -n 1`
$ python $root/cli.py --app=b2gdesktop \
--binary=$b2g \
--profile=profile \
--type=b2g \
--timeout=10000 \
tests/python/gaia-ui-tests/gaiatest/tests/accessibility/lockscreen/test_a11y_un\
lock_to_homescreen.py

The script file can be edited to target specific test files or directories, and add extra command line parameters — specifically the line that runs cli.py.

Important: When using B2G Desktop or an Emulator, try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.

Note: You can enter the deactivate command to exit to the normal console when the tests have finished.

Finding the HTML Report in a completed Travis job

In Travis jobs we produce an HTML Report and upload it to a public webspace.

  1. Open the gaia-ui-tests job for the pull request or commit
  2. Scroll down to "after_script" section and expand the text
  3. Near the end of "after_script", the console outputs a link to the location of the report on Amazon AWS

When the tests are run locally you'll be able to find the report at: gaia/artifacts/gaia_ui_tests_report.html

Note: At the moment this only works for Merge commits, but we hope to use this on pull requests in the future

Testing locally with TBPL configuration for Gecko and Gaia developers

In this section we will focus upon replicating a TBPL test locally. Before testing locally with a TBPL configuration you need to be aware that it builds a Gaia profile separately. The profile that comes packaged with the TBPL build or rel-eng build is NOT the same Gaia profile that TBPL uses.

Prerequisites

  • A B2G desktop build.  You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • The Gaia Github repository checked out (for making the Gaia profile, and installing gaiatest and the test files).
  • A testvars.json file created (as expained in the relevant section below.)

Building a Gaia profile with TBPL's  settings

Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '1',
            'NOFTU': '0'}

These settings can drastically change the way desktop B2G behaves. It is very important to match TBPL's settings.

The next step is to checkout the git commit that matches the hg commit that TBPL ran with. You can also find the hg commit in the "brief log" of the TBPL run but you will need to correlate that with the git commit yourself.

The gaia test repository includes a make script for building a Gaia profile. Navigate to the Gaia repo that you have cloned locally, and build the profile using the environment variables we found on TBPL:

DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=1 NOFTU=0 make

Running the tests

Using the same git commit that TBPL used to run the test, go to the gaia/tests/python/gaia-ui-tests directory and install gaiatest using the following command:

python setup.py develop

Now that you have built the Gaia profile, the only significant difference to running these tests is that you must direct gaiatest to the binary location and profile location. As TBPL runs an isolated environment it uses its own set of tests, set in tbpl-manifest.ini.

Inside the gaia-ui-tests directory, run the below command, with the following substitutions made:

  • $BINARY_PATH: The path to the b2g-bin file in the directory you have built or unpacked desktop B2G to.
  • $PROFILE_PATH: The path to your locally build Gaia profile. If built from the Gaia repository it will be ~/gaia/profile.
gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/tbpl-manifest.ini --type=b2g

Note: There is no need to forward port 2828 unless you have set the Marionette server to use a different default port.

Important: Try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.

Finding the HTML report for a completed TBPL job

All TBPL jobs (both pass and fail) have a HTML report output generated and stored.

  1. For the commit, click Gu.
  2. In the status bar at the bottom of TBPL find the section titled TinderboxPrint: Uploaded output.html.
  3. Copy the link to a browser tab to view the report.

Testing against an already running Desktop B2G binary

In this section we will look at running the tests against an already running Desktop B2G binary with modified source code.

Prerequisites

  • A B2G desktop build with your Gecko changes.  You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
  • Your debugger of choice attached to the B2G binary
  • The Gaia Github repository checked out (for installing gaiatest and the test files).
  • A testvars.json file created (as expained in the relevant section below.)

Running the tests

Running tests against an existing b2g process is quite easy — you've already gone through the hard parts to build B2G!

If you have not already installed gaiatest then do so now by going to gaia/tests/python/gaia-ui-tests and running the following:

python setup.py develop

Once you have started the b2g binary process and attached your debugger you need to direct gaiatest to the port using the --address command. Do not use --binary or --profile, otherwise gaiatest will just send commands directly to the port without attempting to start or close the binary.

gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py

Note: We have omitted the --restart command too. As gaiatest does not attempt to stop or start the binary some data from your test run may be left behind in Firefox OS databases or even in the DOM. It is your responsibility to reset B2G back to a basic state before trying to run the test for a second time.

Test Variables (testvars.json)

The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data . Please read the warning before you set up your test variables file.

We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

When running your tests add the argument: --testvars=(filename).json

How to configure WiFi using testvars.json (for devices)

By setting the Wifi key in testvars.json you can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

No WiFi authentication:

"wifi": { "ssid": "MyNetwork"} 

WEP authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

WPA-PSK authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

Due to Bug 775499, WiFi connections via WPA-EAP are not possible at this time.

 

Troubleshooting

This section details some common error messages, along with advice on how the problems could be fixed.

Common problems before the test has started

Error Message Corresponding Actions
error: [Errno 111] Connection refused

Reissue the "adb forward tcp:2828 tcp:2828" command

B2G process is not running

Element ... not visible before timeout Make sure the element would display on the app you test
TimeoutException: Condition timed out make sure the condition on the app is the same as your expectations
marionette.errors.MarionetteException: localhost:2828 is unavailable. Something is blocking the port, often the `adb forward` command. `adb kill-server` will resolve that, or closing the app that is keepin the port open
ImportError: No module named bluetooth

Replace --type=b2g in your command with --type=b2g-bluetooth
OR
Install pybluez Python bluetooth package
 

OSError: [Errno 2] No such file or directory: '~/moz/gaia/profile' Check to make sure that you have not used ~ on the command line where you meant $HOME.

 

Document Tags and Contributors

Last updated by: chrisdavidmills,
Hide Sidebar