Running Tests

Tests can be run in-tree with a locally built Firefox through mach, as well as with a downloaded Firefox binary against in- and out-of-tree tests with runtests.py.

Running in-tree tests, mach will automatically manage the Python virtual environment in which your tests are run.  The Marionette client that is picked up is the one that is in-tree at testing/marionette/client.

If you want to run tests from a downloaded tests.zip file, you will first need to set up the Python Marionette client. Second, you will need to download the tests from the tests.zip package found attached to Treeherder jobs for your system. If you are using a tests.zip package, extract it, and find the tests under marionette/tests/testing/marionette/client/marionette/tests.

You can launch the tests using one of the methods below. If desired, you can replace the path to unit-tests.ini with the path to a particular test.  See the Marionette Python TestRunner for more options.

For Desktop Firefox

With a local build

You can use mach (German for “do”) to run all the tests:

% ./mach marionette-test

Individual tests can be run by specifying the file name:

% ./mach marionette-test testing/marionette/harness/marionette/tests/unit/test_findelement.py

You can see what extra options are provided using the -h flag:

% ./mach marionette-test -h
usage: mach [global arguments] marionette-test [command arguments]

Run a Marionette test (Check UI or the internal JavaScript using marionette).

Global Arguments:
  -v, --verbose         Print verbose output.
  -l FILENAME, --log-file FILENAME
                        Filename to write log data to.
  --log-interval        Prefix log line with interval from last message rather
                        than relative time. Note that this is NOT execution
                        time if there are parallel operations.
  --log-no-times        Do not prefix log lines with times. By default, mach
                        will prefix each output line with the time since
                        command start.
  -h, --help            Show this help message.
  --debug-command       Start a Python debugger when command is dispatched.

Command Parameters:
  tests                 Tests to run.

Command Arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Increase verbosity to include debug messages with -v,
                        and trace messages with -vv.
  --emulator {x86,arm}  if no --address is given, then the harness will launch
                        a B2G emulator on which to run emulator tests. if
                        --address is given, then the harness assumes you are
                        running an emulator already, and will run the emulator
                        tests using that emulator. you need to specify which
                        architecture to emulate for both cases
  --emulator-binary EMULATOR_BINARY
                        launch a specific emulator binary rather than
                        launching from the B2G built emulator
  --emulator-img EMULATOR_IMG
                        use a specific image file instead of a fresh one
  --emulator-res EMULATOR_RES
                        set a custom resolution for the emulatorExample:
                        "480x800"
  --sdcard SDCARD       size of sdcard to create for the emulator
  --no-window           when Marionette launches an emulator, start it with
                        the -no-window argument
  --logcat-dir LOGDIR   directory to store logcat dump files
  --logcat-stdout       dump adb logcat to stdout
  --address ADDRESS     host:port of running Gecko instance to connect to
  --device DEVICE_SERIAL
                        serial ID of a device to use for adb / fastboot
  --adb-host ADB_HOST   host to use for adb connection
  --adb-port ADB_PORT   port to use for adb connection
  --type TYPE           the type of test to run, can be a combination of
                        values defined in the manifest file; individual values
                        are combined with '+' or '-' characters. for example:
                        'browser+b2g' means the set of tests which are
                        compatible with both browser and b2g; 'b2g-qemu' means
                        the set of tests which are compatible with b2g but do
                        not require an emulator. this argument is only used
                        when loading tests from manifest files
  --homedir HOMEDIR     home directory of emulator files
  --app APP             application to use
  --app-arg APP_ARGS    specify a command line argument to be passed onto the
                        application
  --binary BINARY       gecko executable to launch before running the test
  --profile PROFILE     profile to use when launching the gecko process. if
                        not passed, then a profile will be constructed and
                        used
  --pref PREFS_ARGS     A preference to set. Must be a key-value pair
                        separated by a ':'.
  --preferences PREFS_FILES
                        read preferences from a JSON or INI file. For INI, use
                        'file.ini:section' to specify a particular section.
  --addon ADDON         addon to install; repeat for multiple addons.
  --repeat REPEAT       number of times to repeat the test(s)
  --testvars TESTVARS   path to a json file with any test data required
  --tree TREE           the tree that the revision parameter refers to
  --symbols-path SYMBOLS_PATH
                        absolute path to directory containing breakpad
                        symbols, or the url of a zip file containing symbols
  --timeout TIMEOUT     if a --timeout value is given, it will set the default
                        page load timeout, search timeout and script timeout
                        to the given value. If not passed in, it will use the
                        default values of 30000ms for page load, 0ms for
                        search timeout and 10000ms for script timeout
  --startup-timeout STARTUP_TIMEOUT
                        the max number of seconds to wait for a Marionette
                        connection after launching a binary
  --shuffle             run tests in a random order
  --shuffle-seed SHUFFLE_SEED
                        Use given seed to shuffle tests
  --total-chunks TOTAL_CHUNKS
                        how many chunks to split the tests up into
  --this-chunk THIS_CHUNK
                        which chunk to run
  --sources SOURCES     path to sources.xml (Firefox OS only)
  --server-root SERVER_ROOT
                        url to a webserver or path to a document root from
                        which content resources are served (default:
                        /Users/dburns/development/mozilla-
                        central/testing/marionette/harness/marionette/www).
  --gecko-log GECKO_LOG
                        Define the path to store log file. If the path is a
                        directory, the real log file will be created given the
                        format gecko-(timestamp).log. If it is a file, if will
                        be used directly. '-' may be passed to write to
                        stdout. Default: './gecko.log'
  --logger-name LOGGER_NAME
                        Define the name to associate with the logger used
  --jsdebugger          Enable the jsdebugger for marionette javascript.
  --pydebugger PYDEBUGGER
                        Enable python post-mortem debugger when a test fails.
                        Pass in the debugger you want to use, eg pdb or ipdb.
  --socket-timeout SOCKET_TIMEOUT
                        Set the global timeout for marionette socket
                        operations.
  --e10s                Enable e10s when running marionette tests.
  --tag TEST_TAGS       Filter out tests that don't have the given tag. Can be
                        used multiple times in which case the test must
                        contain at least one of the given tags.
  --workspace WORKSPACE
                        Path to directory for Marionette output. (Default: .)
                        (Default profile dest: TMP)

If you are developing the Marionette server component, it is often useful to redirect Gecko’s standard output to your shell’s, as such:

% ./mach marionette-test TEST --gecko-log -

Similarily the --jsdebugger flag is occasionally useful in order to attach a Developer Tools Browser Debugger to the Marionette session.

With a downloaded desktop Firefox build and tests.zip

Activate the virtualenv that you installed Marionette in (if you used one), then:

cd /path/to/extracted/tests.zip/marionette/marionette
python runtests.py --binary=/path/to/firefox-executable --type=browser ../tests/testing/marionette/client/marionette/tests/unit-tests.ini

For Firefox OS

With a device build

Activate the virtualenv that you installed Marionette in (if you used one), make sure your device is connected via adb, then:

adb forward tcp:2828 tcp:2828
cd testing/marionette/client/marionette
python runtests.py --address localhost:2828 --type=b2g tests/unit/unit-tests.ini

With an emulator build

Activate the virtualenv that you installed Marionette it (if you used one), make sure your device is connected via adb, then:

cd testing/marionette/client/marionette
python runtests.py --emulator arm --homedir $B2G_DIR --type=b2g tests/unit/unit-tests.ini

B2G_DIR is either the path to your B2G directory (if you've built your own emulator), or the path to a downloaded and extracted emulator package.

With a B2G desktop build

The only tests that are supported right now for B2G Desktop build are gaia-ui-tests. See that page for instructions. However, do note that these tests are being phased out in favour of Node.js-backed Gaia integration tests.

Document Tags and Contributors

 Last updated by: GijsKruitbosch,