Your Search Results

    Gaia UI Tests: Running Tests

    This article provides a detailed rundown of how to run the actual gaia-ui-tests suite that resides inside Gaia itself, on B2G Desktop, and real Firefox OS devices. The Gaia project uses Treeherder to do per-commit continuous integration and pull request tests.

    The gaia-ui-tests have been developed with the capability to run on a real device device, emulators, and the B2G Desktop client. By far the easiest method is to run against B2G Desktop. Please note that you should choose the right branch of Gaia to run against the version of Firefox OS that you are using.

    Testing on B2G Desktop

    The B2G desktop client is as it sounds: it's a desktop version of B2G that you can use to perform web app tests on Firefox OS from the comfort of your desktop. If you are not familiar with it, you can find more out by reading Using the B2G desktop client. This section will show you how to run gaia-ui-tests on B2G desktop.


    • A B2G desktop build. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
    • The Gaia Github repository cloned and checked out. You will also need to generate a profile. Note that you must not pass DEBUG=1 while generating the profile - using a debug gaia profile will cause gaia-ui-tests to fail to connect with the b2g desktop instance.
    • A testvars.json file created (as explained in the relevant section below).

    The desktop B2G client is ideal for developing tests when a real device is unavailable and there is no need for phone-level functions. It is fast, accessible, and will run on Linux, Mac, and Windows!

    Running the tests

    We'll look at running new tests; we'll install the in-tree version of gaiatest directly from the Gaia repository so that our changes are picked up immediately. In your cloned version of Gaia, navigate to gaia/tests/python/gaia-ui-tests and run the setup command:

    python develop

    This might give permission denied errors when not running it inside a virtual environment. In that case, install virtualenv and virtualenvwrapper, if you haven't installed those before:

    pip install virtualenv
    pip install virtualenvwrapper (perhaps need to use sudo here)
    export WORKON_HOME=~/Envs
    source /usr/local/bin/
    echo "source /usr/local/bin/" >> .bash (rc or _profile)

    Then, create and enter that virtual environment:

    mkvirtualenv gaia-ui-test

    Via Marionette, gaiatest is able to launch the B2G Desktop when we start the test. To do this, you need to pass in the path to the B2G binary file and B2G profile. In the below example we have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped B2G Desktop to after you downloaded it. The command to run all the tests is:

    gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart --type=b2g ./gaiatest/tests/functional/manifest.ini

    Tests that are incompatible with the B2G Desktop client — for example those that make calls and send SMS — are automatically skipped from the test run.

    You can run a test directly by calling the file on the command line:

    gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars.json --restart ./gaiatest/tests/functional/clock/

    Testing on Firefox OS Devices

    Testing on real Firefox OS devices is more complicated, but it gives by far the most accurate results, with access to all the real device APIs, etc. If you don't have a real Firefox OS device, emulator works in some functions. You can use emulator to test functionality as well.


    Running the tests

    The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the gaia/tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

    python develop

    Note: Before you start to run the test, read through this warning in order to acknowledge that running gaia-ui-tests on a device may cause data to be deleted from the device!

    Debug Firefox OS builds: Firefox OS builds built with DEBUG=1 will need a greater timeout than the default (20 seconds). Try adding to the command line --timeout 60000 or higher, depending upon the speed of the device.

    Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb. Run the following command:

    adb forward tcp:2828 tcp:2828

    Please be reminded that you'll need to start your emulator before the command, if you are not running on a real Firefox OS device.

    You can use the following command to execute all the gaia-ui-tests:

    gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

    Or use the following if you want to run a single test (substituting the last part for the actual test you want to run):

    gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/tests/functional/clock/

    After the test, you will see all the pass/fail results and stacktraces for each file.

    Testing locally with Treeherder configuration for Gecko and Gaia developers

    In this section we will focus upon replicating a Treeherder test locally. Before testing locally with a Treeherder configuration, you need to be aware that it builds a Gaia profile separately. The profile that comes packaged with the Treeherder build or rel-eng build is NOT the same Gaia profile that Treeherder uses.


    • A B2G desktop build. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
    • The Gaia Github repository checked out (for making the Gaia profile, and installing gaiatest and the test files).
    • A testvars.json file created (as explained in the relevant section below).

    Building a Gaia profile with Treeherder's  settings

    Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

    Using env: {'DEBUG': '0',
                'DESKTOP': '0',
                'DESKTOP_SHIMS': '0',
                'NOFTU': '0'}

    These settings can drastically change the way desktop B2G behaves. It is very important to match Treeherder's settings.

    The next step is to checkout the git commit that matches the hg commit that Treeherder ran with. You can also find the hg commit in the "brief log" of the Treeherder run but you will need to correlate that with the git commit yourself.

    The gaia test repository includes a make script for building a Gaia profile. Navigate to the Gaia repo that you have cloned locally, and build the profile using the environment variables we found on Treeherder:


    Running the tests

    Using the same git commit that Treeherder used to run the test, go to the gaia/tests/python/gaia-ui-tests directory and install gaiatest using the following command:

    python develop

    Now that you have built the Gaia profile, the only significant difference to running these tests is that you must direct gaiatest to the binary location and profile location.

    Inside the gaia-ui-tests directory, run the below command, with the following substitutions made:

    • $BINARY_PATH: The path to the b2g-bin file in the directory you have built or unpacked desktop B2G to.
    • $PROFILE_PATH: The path to your locally build Gaia profile. If built from the Gaia repository it will be ~/gaia/profile.
    gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/manifest.ini --type=b2g

    Note: There is no need to forward port 2828 unless you have set the Marionette server to use a different default port.

    Important: Try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.

    Finding the HTML report for a completed Treeherder job

    All Treeherder jobs (both pass and fail) have an HTML report output generated and stored.

    1. For the commit and Gip job, click the chunk of interest (e.g., f1, f2).
    2. In the status bar at the bottom of the screen find the section titled: output.html.
    3. Click link to view the report.

    Testing against an already running Desktop B2G binary

    In this section we will look at running the tests against an already running Desktop B2G binary with modified source code.


    • A B2G desktop build with your Gecko changes. You can either download it (including from a Try push) and unpack to a directory on your computer or virtual machine or compile it yourself, with the addition of ENABLE_MARIONETTE=1 in your mozconfig.
    • Your debugger of choice attached to the B2G binary.
    • The Gaia Github repository checked out (for installing gaiatest and the test files).
    • A testvars.json file created (as explained in the relevant section below).

    Running the tests

    Running tests against an existing b2g process is quite easy — you've already gone through the hard parts to build B2G!

    If you have not already installed gaiatest then do so now by going to gaia/tests/python/gaia-ui-tests and running the following:

    python develop

    Once you have started the b2g binary process and attached your debugger you need to direct gaiatest to the port using the --address command. Do not use --binary or --profile, otherwise gaiatest will just send commands directly to the port without attempting to start or close the binary.

    gaiatest --testvars testvars.json gaiatest/tests/functional/

    Note: We have omitted the --restart command too. As gaiatest does not attempt to stop or start the binary, some data from your test run may be left behind in Firefox OS databases or even in the DOM. It is your responsibility to reset B2G back to a basic state before trying to run the test for a second time.

    Test Variables (testvars.json)

    The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data. Please read the warning before you set up your test variables file.

    We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

    To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

    When running your tests add the argument: --testvars=(filename).json

    How to configure WiFi using testvars.json (for devices)

    By setting the Wifi key in testvars.json you can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

    No WiFi authentication:

    "wifi": { "ssid": "MyNetwork"} 

    WEP authentication:

    "wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

    WPA-PSK authentication:

    "wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

    Due to Bug 775499, WiFi connections via WPA-EAP are not possible at this time.



    This section details some common error messages, along with advice on how the problems could be fixed.

    Common problems before the test has started

    Error Message Corresponding Actions
    error: [Errno 111] Connection refused

    Reissue the "adb forward tcp:2828 tcp:2828" command

    B2G process is not running

    Element ... not visible before timeout Make sure the element would display on the app you test
    TimeoutException: Condition timed out make sure the condition on the app is the same as your expectations
    marionette.errors.MarionetteException: localhost:2828 is unavailable. Something is blocking the port, often the `adb forward` command. `adb kill-server` will resolve that, or closing the app that is keeping the port open
    ImportError: No module named bluetooth

    Replace --type=b2g in your command with --type=b2g-bluetooth
    Install pybluez Python bluetooth package

    OSError: [Errno 2] No such file or directory: '~/moz/gaia/profile' Check to make sure that you have not used ~ on the command line where you meant $HOME.


    Document Tags and Contributors

    Last updated by: kscarfone,
    Hide Sidebar