You've just written a feature and (hopefully!) want to test it. Or you've decided that an existing feature doesn't have enough tests and want to contribute some. But where do you start? You've looked around and found references to things like "xpcshell" or "mozmill" or "talos". What code, features or platforms do they all test? Where do their feature sets overlap? In short, where should your new tests go? This document is a starting point for those who want to start to learn about Mozilla's automated testing tools and procedures. Below you'll find a short summary of each framework we use, and some questions to help you pick the framework(s) you need for your purposes.
Note: If you read this article and still have questions, try reading some of the further links provided below for more information, and hop on to the #ateam or #qa IRC channels or Mozilla Forums to ask questions.
These tests are found within the mozilla-central tree, along with the product code. They are all run when a changeset is pushed to mozilla-central, mozilla-inbound, or try, with the results showing up on Treeherder. They can also be run on their own.
The tests are run on machines in a very large pool. For the most part, all tests of a particular type run on the same pool of machines, regardless of branch. One substantial exception is that try builds are performed on a pool isolated from all other builds.
Note: refer also to how test harnesses work for more information on how these tests are run.
|Platform||Process||Environment||Privilege||What is tested|
||Crashtest||HTML||Yes||Yes||–||–||Yes||Content||Yes||Yes||–||Crashes, assertions, or leaks when opening a web page.|
||Reftest||HTML||Yes||Yes||–||–||Yes||Content||Yes||Yes||–||Layout and graphics correctness (two web pages are rendered identically.)|
||IPC||?||–||–||–||?||?||?||?||?||?||Plugin APIs, particularly out-of-process plugins.|
|JS||Yes||–||–||Yes||Allow||Browser||Yes||–||Yes||How the browser UI interacts with itself and with content.|
||Mochitest devtools||JS||Yes||–||–||Yes||Allow||Browser||Yes||–||Yes||Firefox developer tools. Based on Mochitest browser-chrome.|
||gaia-build-integration||?||–||–||Yes||?||?||?||?||?||?||The Gaia build system.|
||gaia-ui-tests||Python||–||–||Yes||?||?||?||?||?||?||Gaia integration. Marionette- and Python- based.|
||Marionette||?||?||?||?||?||?||?||?||?||?||Large scope tests like communication with multiple remote Gecko processes.|
|?||?||?||?||?||?||WebAPIs. This takes advantage of the Firefox OS emulator's ability to virtualize much of the hardware on a B2G device.|
||Jetpack||JS||?||?||?||?||?||?||?||?||?||Add-on SDK code included in Firefox. If you need help interpreting test failures or fixing bugs in the SDK code, talk to the SDK team in #jetpack.|
|–||–||?||?||?||?||?||?||Memory-related errors, such as heap block overflows/underflows, use of uninitialized memory, bad frees, and memory leaks.|
- Abbreviation for the test suite used by Treeherder. The first letter generally indicates which of the test harnesses is used to execute the test. The letter in parentheses identifies the actual test suite.
- Common name used when referring to the test suite.
- File type
- When adding a new test, you will generally create a file of this type in the source tree and then declare it in a manifest or makefile.
Most test suites are supported only on a subset of the available plaforms and operating systems. Unless otherwise noted:
- Desktop tests run on Windows, Mac OS X, and Linux.
- Mobile tests run on Android emulators or remotely on Android devices.
- B2G tests run on B2G emulators or remotely on B2G devices.
- When Parent is indicated, the test file will always run in the parent process, even when the browser is running in Electrolysis (e10s) mode.
- When Child is indicated, the test file will run in the child process when the browser is running in Electrolysis (e10s) mode.
- The Allow label indicates that the test has access to mechanisms to run code in the other process.
- The Content indication means that the test is run inside a content page loaded by a browser window.
- The Browser Profile column indicates whether a browser profile is loaded when the test starts. The Allow label means that the test can optionally load a profile using a special command.
Talos is the framework used for performance testing.
|Symbol||Name||Platform||What is tested|
||tp||Yes||Yes||-||The load time of a set of test web pages taken from the Alexa top 500, with the full UI enabled.|
||svg||Yes||Yes||-||SVG rendering time.|
||chrome||Yes||-||-||Various suites with the full UI enabled.|
||dirty||Yes||-||-||A populated history database (places.sqlite) that more closely resembles the typical viewing habits of an average user.|
||nochrome||Yes||-||-||Various suites with the full UI disabled.|
||tp nochrome||-||Yes||-||The load time of a set of test web pages taken from the Alexa top 500, with the full UI disabled.|
||tspaint||-||Yes||-||The startup time of Firefox, by opening the browser 20 times.|
||tsvgx||-||Yes||-||SVG rendering time.|
||tcanvasmark||-||Yes||-||Canvas animation, using the third-party CanvasMark benchmark.|
||tcheckerboard2||-||Yes||-||The amount of checkerboarding (delayed painting), by loading a test page then zooming and panning. Robocop-based.|
||trobopan||-||Yes||-||The lag time in rendering a web page by loading it and panning to the bottom and back to the top. Robocop-based.|
||tprovider||-||Yes||-||The time taken to query the Android OS history database with thousands of entries. Robocop-based.|
Not in production
Mozmill is an extensive framework for browser automation. It is an extension with an extensive API to help you write functional tests that simulate user interactions, as well as a full unit test API. It can be used to test configurations that are difficult to simulate in buildbot automation. QA automation uses Mozmill to test localized builds, performance over time, and other scenarios. Mozmill tests are currently not run in buildbot and do not report to Treeherder.
So which should I use?
Here's a series of questions to ask about your work when you want to write some tests.
Is it low-level code?
Does it cause a crash?
If you've found pages that crash Firefox, add a crashtest to make sure future versions don't experience this crash (assertion or leak) again. Note that this may lead to more tests once the core problem is found.
Is it a layout/graphics feature?
Reftest is your best bet, if possible.
Do you need to verify performance?
Are you testing UI features?
Are you testing Mobile/Android?
For Mobile UI, look at Robocop. There are some specific features that Mochitest or Reftest can cover. Mochitest-chrome and browser-chrome do not run on Android. If you want to test performance, Talos runs fine with a few limitations (use the
--noChrome options) and smaller cycles (e.g. 10 iterations instead of 20, etc.)
Are you doing none of the above?
- To get your tests run through buildbot, try Mochitest, or, if higher privileges are required and you don't need mobile testing, try Mochitest chrome tests.
- While not in buildbot automation, Mozmill is a bigger framework that can test almost anything, on the desktop at least.
- For Desktop Firefox or B2G, or if you just want to see the future of Gecko testing, look into the on-going Marionette project.
Need to get more data out of your tests?
Most test jobs now expose an environment variable named
$MOZ_UPLOAD_DIR. If this variable is set during automated test runs, you can drop additional files into this directory, and they will be uploaded to a web server when the test finishes. The URLs to retrieve the files will be output in the test log.
Need to set preferences for test-suites?
Sadly, the locations for preferences are all over the place, some are listed below - bug 1023483 has more details and locations.
- mochitests: http://dxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js
- xpcshell: http://dxr.mozilla.org/mozilla-central/source/testing/xpcshell/head.js
- global: https://hg.mozilla.org/build/talos/file/tip/talos/PerfConfigurator.py#l251
- per-test in test.py, e.g.: https://hg.mozilla.org/build/talos/file/49b74c08dad4/talos/test.py#l209
- the current Talos revision is pointed to here: testing/talos/talos.json#l8
- JS engine tests: http://mxr.mozilla.org/mozilla-central/source/js/src/tests/user.js
Note: The previous page was largely out of date. The new page is intended as a guide to the testing frameworks at Mozilla; some of the old page should be split out to a "Running Automated Tests" page to complement the "Developing Tests" page (which itself needs to be updated).
Also see the "Continuous Integration" page.