Running Tests

Marionette-based tests can be run in-tree with a locally built Firefox through mach marionette-test, as well as with a downloaded Firefox binary against in- and out-of-tree tests with runtests.py. The mach command calls runtests.py on the binary built from your mozilla-central checkout. All the same options should be available in both cases. (Aside: Are you looking to run unit tests for the Marionette Test Runner? That's described over in Developer setup under "Marionette Harness Unit Tests")

Running in-tree tests, mach will automatically manage the Python virtual environment in which your tests are run.  The Marionette client that is picked up is the one that is in-tree at testing/marionette/client.

If you want to run tests from a downloaded tests.zip file, you will first need to set up the Python Marionette client. Second, you will need to download the tests from the tests.zip package found attached to Treeherder jobs for your system. If you are using a tests.zip package, extract it, and find the tests under marionette/tests/testing/marionette/client/marionette/tests.

You can launch the tests using one of the methods below. If desired, you can replace the path to unit-tests.ini with the path to a particular test.  See the Marionette Python TestRunner for more options.

For Desktop Firefox

With a local build

Assuming you've run ./mach build successfully, you can use mach (German for “do”) to run all the tests:

% ./mach marionette-test

Individual tests can be run by specifying the file name:

% ./mach marionette-test testing/marionette/harness/marionette/tests/unit/test_findelement.py

You can see what extra options are provided using the -h flag.

If you are developing the Marionette server component, it is often useful to redirect Gecko’s standard output to your shell’s, as such:

% ./mach marionette-test TEST --gecko-log -

Similarily the --jsdebugger flag is occasionally useful in order to attach a Developer Tools Browser Debugger to the Marionette session.

With a downloaded desktop Firefox build and tests.zip

Activate the virtualenv that you installed Marionette in (if you used one), then:

cd /path/to/extracted/tests.zip/marionette/marionette
python runtests.py --binary=/path/to/firefox-executable ../tests/testing/marionette/client/marionette/tests/unit-tests.ini

For Firefox for Android (Fennec)

With a local build

Assuming you've run ./mach build, ./mach package and ./mach install successfully (see Simple Firefox for Android build), you can use mach (German for “do”) to run all the tests. As a result of setting up a Fennec build, you will have Android SDK set up, likely somewhere under ~/.mozbuild -- this will be useful shortly.

The Marionette Test Runner can take care of launching an Android emulator and a Marionette-enabled Fennec for you, but it needs to know where to find an emulator and an AVD. You can use mach commands to set up these pieces.

# This command will offer to download and set up an AVD, and use it to launch an emulator 
# The AVD is likely set up at ~/.mozbuild/android-device/avd
./mach android-emulator
# Install Fennec on the emulator
./mach install
# Run Fennec at least once to set up a profile
./mach run

To launch an emulator and launch Fennec in it and run tests:

./mach marionette-test --emulator --app='fennec' --avd-home /path/to/.mozbuild/android-device/avd --emulator-binary /path/to/.mozbuild/android-sdk-macosx/tools/emulator

If you set an AVD_HOME environment variable and if emulator command is in your PATH, you may omit the --avd-home and --emulator-binary arguments.

To connect to an already-running emulator and launch Fennec:

./mach marionette-test --emulator --app='fennec' --address=localhost:2828 

To  connect to an already-running Fennec instance, you will need to enable Marionette manually (by building with ENABLE_MARIONETTE=1 and enabling marionette in the browser profile).

adb forward tcp:2828 tcp:2828
./mach marionette-test --address=localhost:2828

 

Document Tags and Contributors

 Last updated by: maja_zf,