web-platform-tests is a regression test suite covering standard Web platform features. It is shared both among browser vendors and across browser platforms (such as Mozilla's Gecko and Servo), and is one of the primary means of verifying that browser implementations correctly implement the standards and are interoperable. Because of this, adding new tests to web-platform-tests (as opposed to Mozilla-specific Reftests, Mochitests, etc. suites) is encouraged whenever writing tests that might sensibly be run in multiple browsers.
Mozilla has integrated web-platform-tests into mozilla-central, and has a two-way sync mechanism that allows developers to add/modify tests directly in the testing/web-platform directory in the Gecko source. (Changes are regularly synchronized between the Gecko source and the upstream GitHub repository.)
Like other test suites, web-platform-tests are invoked using the
Individual tests can either be run by path in the source tree or by name (i.e. path under the server root). For example, the following two commands perform the same test:
./mach wpt testing/web-platform/tests/dom/historical.html ./mach wpt /dom/historical.html
One can also pass the path to one or more directories to run all tests under that directory.
Running under a debugger
The tests can be run under a debugger using the same flags as for other test suites. For example:
--pause-on-unexpected flag can also be useful to pause on the first failing test allowing you to attach a debugger and reload the test to debug it.
web-platform-tests use structured logging, and all the usual structured logging options are available.
General guides to the API for writing web-platform-tests are found on Test the Web Forward. However, there are some Mozilla-specific concerns detailed below.
All web-platform-tests which are upstreamed live in
testing/web-platform/tests. Since all files in this directory are synced with upstream, it is important that only tests that have a pass condition matching the specification and are suitable for running in multiple browsers are checked in here. Tests that cannot (yet) be upstreamed, such as those which use Gecko-specific features or are not in a state where they match the spec, can be checked in to
/_mozilla/ prepended to their URL.
web-platform-tests allows mixing multiple test types in a single directory (e.g. both reftests and testharness.js tests; wdspec tests are an exception as they must live under
webdriver/). Unlike Mozilla reftests or mochitests, there is no human-edited manifest file identifying tests; instead tests are identified by their contents, and the information is cached in
MANIFEST.json files checked in under
testing/web-platform/mozilla/meta. After adding or updating a test these manifests must be updated, the simplest way to do this is to run
Alternatively, the tests may be run with the
--manifest-update flag e.g.
mach wpt --manifest-update testing/web-platform/tests/spec/new_test.html
Creating new tests
New web-platform-tests can be created just by creating the relevant files and then running the manifest-update command above. To help with the process, the
mach wpt-create command is able to prefill files with test boilerplate. In order to create a new testharness.js test, run
./mach wpt-create testing/web-platform/tests/spec/new_test.html
This will create a test file with the
testharness.js boilerplate, open it in your local
$EDITOR, if defined, and then update the test manifest after the editor process terminates.
In order to create a new reftest, the same procedure as above may be followed, but passing the
--reftest arguments. In addition, the path to the reference file may be provided using the
--reference argument e.g.
./mach wpt-create -r testing/web-platform/tests/spec/new_test.html --reference testing/web-platform/tests/spec/new_test_ref.html
If the reference doesn't exist it will be opened in the editor in addition to the test file.
Because web-platform-test files are synced with upstream and do not use a human-written manifest, it's necessary to keep implementation-specific metadata outside the tests themselves. This is done in a metadata file under
testing/web-platform/meta with the same path as the test, but an
.ini extension e.g.
testing/web-platform/tests/dom/historical.html has a metadata file under
testing/web-platform/meta/dom/historical.html.ini. In addition, metadata can be set for a whole subtree of tests using a metadata file called
__dir__.ini under the metadata root of that subtree (e.g.
.ini extension, these files are not true ini files, since they allow nesting and conditional logic on values.
Because web-platform-tests are intended to check the standard behavior rather than the behavior of specific implementations, it is common for tests to fail in specific implementations. Therefore, the expected result of each test that doesn't pass is kept in the corresponding metadata file. A simple example of such a file might be:
[Subtest name for failing test]
[Subtest name for erroring test]
Expectations can be made platform-specific using a simple set of python-like conditions. For example, for a test that times out on Linux and fails on other platforms:
if os == "linux": TIMEOUT
The available set of condition variables is that provided by mozinfo plus a boolean variable
e10s that is true when e10s is enabled.
Expectation data can be updated automatically on the basis of a test run (e.g. from try), using the raw structured log files. On treeherder, these are files listed as artifact uploaded:
wpt_raw.log. Locally they can be generated using a
mach command like:
./mach wpt testing/web-platform/tests/dom/historical.html --log-raw=historical.log
Then to update the expectation data:
./mach wpt-update historical.log
By default, this will make a commit (in a git tree) or an mq patch (in an hg tree). To override this behavior, use
--no-patch. If the expected result of a test is platform-specific, you must provide multiple logs—one for each platform—to the update command, or the correct metadata will not be generated. If you want to wipe away the existing metadata for the test rather than try to update the conditions in place (such as when a test goes from having platform-specific behavior to generic behavior) use
You can disable a test by adding a
disabled key to the metadata file. The value can be anything except the special values
@Reset, but by convention is the bug number detailing the reason the test was disabled. For example:
if os == "win": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567
Per-test prefs can be set using a metadata item called
prefs which takes a list of preferences to set and the values to assign to them. This lets tests run against features which are preferenced off by default, for instance.
__dir__.ini file creates metadata that applies to the entire subtree beneath it. For example, we can disable all DOM tests using a file in
Having done this, we might need to re-enable a specific test under
dom/. This can be done using the special value
@False in the metadata file for that test:
[filename.html] disabled: @False
With lists of preferences, the preferences applied are typically those for the test and those from any
__dir__.ini files up to the test root. If we need to apply a different set of prefs to a specific test, the special value
@Reset prevents all inheritance of prefs from further up the tree.
Running tests in other browsers
Tests can be run in other browsers by using the
--product argument to
mach wpt e.g.
./mach wpt --product chrome testing/web-platform/dom/historical.html
Currently supported values for
edge. For Chrome and Edge the product-specific WebDriver binary is required; for Chrome the mach command will prompt to download this, but for Edge you will have to download the correct WebDriver for your specific Edge version. Since expectation metadata is not shared across products, the default metadata directory for non-Firefox browsers is
testing/web-platform/product/<product>/. This directory is part of the VCS ignore list, so you are free to add any expectation data you find helpful.
In addition to support for running via
mach wpt in a gecko checkout, Servo has its own import of web-platform-tests, with tests living under
tests/wpt/web-platform-tests/. It also has
mach commands similar to those provided in the Gecko build system, with slightly different names. In particular, to run the in-tree copy one can do this:
in a Servo checkout.
web-platform-tests has associated lints, that check for two things:
- Test file correctness/style issues
- Manifest correctness
The lints can be run using mozlint e.g.
mach lint -l wpt -l wpt_manifest
This lint tests for common mistakes made when writing web-platform-tests and style issues.
Examples of problems detected using this lint include:
- Incorrect links to
<meta name=timeout content=long>appearing in an incorrect place in the file
- Trailing whitespace in test files
- Use of windows-style line endings
In rare cases it may be necessary to allow some code that this lint would ordinarily reject (e.g. a test that relies on a literal CR LF pair in the source). In this case the file
testing/web-platform/tests/lint.whitelist must be updated with the error code and the path that should be excluded from linting.
This lint checks that the wpt manifest is up to date. If this lint fails the fix is always to run:
and commit the resulting changes to