The external-media-tests is a Marionette-based test suite designed to test playback of video elements on arbitrary web pages. Right now, tests using this framework play videos on YouTube and Netflix pages, but the basic tests should be usable for any website which plays back HTML5 video. MediaTestCase uses the Firefox Puppeteer library.
Information about the use of external-media-tests in Mozilla automation can be found on wiki.mozilla.org.
Normally, you get this source by cloning a Firefox repo such as mozilla-central. The path to these tests would be in <mozilla-central>/dom/media/test/external, and these instructions refer to this path as
$PROJECT_HOME. You can also get these tests by downloading one of the
tests.comon.zip files matching a downloaded Firefox build from http://archive.mozilla.org, or using
mozdownload with the
--extension tests.common.zip parameters.
Running from a build
If you have built Firefox using
./mach build from a source tree such as mozilla-central, you can run the following command:
$ ./mach external-media-tests
You can pass any of the test options on this command line. They are listed below.
Running with an installer and a tests payload
If you are testing a version of Firefox that you have not built, you must set up a virtualenv to run tests from. You will need a path to the installer or binary of Firefox.
- Create a virtualenv called
$ virtualenv foo $ source foo/bin/activate #or `foo\Scripts\activate` on Windows
external-media-testsin development mode. (To get an environment that is closer to what is used in Mozilla's automation jobs, run
pip install -r requirements.txtfirst.)
$ python setup.py develop ``` Now `external-media-tests` should be a recognized command. Try `external-media-tests --help` to see if it works.
Running the Tests
In the examples below,
$FF_PATH is a path to a recent Firefox binary. If you are running from a source build, the commands below should be invoked with:
$ ./mach external-media-tests
If you are running with a virtualenv, you will need to run like this:
$ external-media-tests --binary $FF_PATH
$ external-media-tests --installer $FF_INSTALLER_PATH
$ external-media-tests --installer-url <url to installer package>
The following examples assume that you will use one of these command lines instead of
$EXTERNAL-MEDIA-TESTS. This runs all the tests listed in
You can also run all the tests at a particular path:
$ $EXTERNAL-MEDIA-TESTS some/path/foo
Or you can run the tests that are listed in a manifest file of your choice:
$ $EXTERNAL-MEDIA-TESTS some/other/path/manifest.ini
By default, the URLs listed in
external_media_tests/urls/default.ini are used for the tests, but you can also supply your ini file of URLs:
$ $EXTERNAL-MEDIA-TESTS --urls some/other/path/my_urls.ini
Running EME Tests
If you are running EME tests in a local build, you must build with the following parameters in your
mozconfig file before you build Firefox:
You must also copy the files
voucher.bin from a production build into your
<objdir>/dist/bin directory. This is not necessary when running production builds.
To run EME tests, you must use a Firefox profile that has the credentials of the service you are using stored in it. With Netflix, this will be created when you log in and save the credentials. You must also use a custom .ini file for URLs to the provider's content and indicate which test to run, as above. For example:
$ $EXTERNAL-MEDIA-TESTS some/path/tests.ini --profile custom_profile_dir --urls some/path/provider-urls.ini
Note, the profile argument needs to specify the directory of a profile, not the name of that profile. For example, the following arguments could be used:
$EXTERNAL-MEDIA-TESTSdom/media/test/external/external_media_tests/playback/eme.ini --profile ~/AppData/Roaming/Mozilla/Firefox/profiles/6usy8n4x.mozilla-debug-netflix/ --urls dom/ media/test/external/external_media_tests/urls/netflix/default.ini
Running tests in a way that provides information about a crash
What if Firefox crashes during a test run? You want to know why! To report crash data, the test runner needs access to a
minidump_stackwalk binary and a
- Download a
minidump_stackwalkbinary for your platform (save it wherever). Get it from Breakpad.
$ chmod +x path/to/minidump_stackwalk # Linux and Mac
- Create an environment variable called
MINIDUMP_STACKWALKthat points to that local path:
$ export MINIDUMP_STACKWALK=path/to/minidump_stackwalk
- Download the
crashreporter-symbols.zipfile for the Firefox build you are testing and extract it. For example, ftp://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-aurora-win32/1427442016/firefox-38.0a2.en-US.win32.crashreporter-symbols.zip
- Run the tests with a
$ $EXTERNAL-MEDIA-TESTS --symbols-path path/to/example/firefox-38.0a2.en-US.win32.crashreporter-symbols
To check whether the above setup is working for you, trigger a (silly) Firefox crash while the tests are running. One way to do this is with the crashme add-on: you can add this to Firefox even while the tests are running. Another way on Linux and Mac OS systems:
- Find the process id (PID) of the Firefox process being used by the tests.
$ ps x | grep 'Firefox'
- Kill the Firefox process with SIGABRT.
# 1234 is an example of a PID $ kill -6 1234
Somewhere in the output produced by
$EXTERNAL-MEDIA-TESTS, you should see something like:
0:12.68 CRASH: MainThread pid:1234. Test:test_basic_playback.py TestVideoPlayback.test_playback_starts. Minidump anaylsed:False. Signature:[@ XUL + 0x2a65900] Crash dump filename: /var/folders/5k/xmn_fndx0qs2jcpcwhzl86wm0000gn/T/tmpB4Bolj .mozrunner/minidumps/DA3BB025-8302-4F96-8DF3-A97E424C877A.dmp Operating system: Mac OS X 10.10.2 14C1514 CPU: amd64 family 6 model 69 stepping 1 4 CPUs Crash reason: EXC_SOFTWARE / SIGABRT Crash address: 0x104616900 ...
Setting up for network shaping tests (browsermobproxy)
- Download the browsermob proxy zip file from http://bmp.lightbody.net/. The most current version as of this writing is
- Unpack the .zip file.
- Verify that you can launch browsermobproxy on your machine by running
browsermob-proxy.baton Windows) on your machine. You may have to do a lot of work to install and use a Java that browsermobproxy would like.
- Import the certificate into your Firefox profile. Select
Preferences->Advanced->Certificates->View Certificates->Import... Navigate to
cybervilliansCA.cer. Select all of the checkboxes.
- Tell Marionette where browsermobproxy is and what port to start it on. Add the following command-line parameters to your external-media-tests command line:
--browsermob-script <browsermob>/bin/browsermob-proxy --browsermob-port 999 --profile <your saved profile>
You can then call browsermob to shape the network. You can find an example in
external_media_tests/playback/test_playback_limiting_bandwidth.py. Another example can be found at https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/unit/test_browsermobproxy.py.
A warning about video URLs
The ini files in
external_media_tests/urls may contain URLs pulled from Firefox crash or bug data. Automated tests don't care about video content, but you might: visit these at your own risk and be aware that they may be NSFW. We do ever not intend to moderate or filter these URLs.
Writing a test
- Tests go into
- Your test should inherit from
MediaTestCase(or a subclass). You may also need to inherit from
VideoPlaybackTestsMixin, which gives you
- You can check
$PROJECT_HOME/external_media_tests/media_utilsfor other useful methods and tests to inherit.
- If you have specific URLs you wish to test, create a
.inifile and place it in
$PROJECT_HOME/external_media_tests/urls, and pass it to your test via the '
--urls' command-line parameter.
- If you wish to create a suite of tests for a particular site or feature, you can list the tests you want into a
.inifile in the
$PROJECT_HOME/external_media_tests/playbackfolder, and pass that file in your test command-line.
API testing for the classes in external-media-tests are on readthedocs.org.
Generating API documentation
The document structure is located in
$PROJECT_HOME/docs. The .rst files are written for Sphinx to process. Please consult the Sphinx documentation if you need to add more sections to the .rst files we already have.
If you make changes to the docstrings in the python code, or if you make changes to the .rst files, you need to test your changes and see how they look. Follow these steps:
- Setup a virtualenv and activate it.
virtualenv docenv . docenv/bin/activate
- Install Sphinx into your virtualenv.
pip install sphinx
- Install External Media Tests into your virtualenv.
cd $PROJECT_HOME python setup.py install
- Run the command to generate the docs
cd $PROJECT_HOME/docs make html
- Examing the resulting html files. The top-level html file will be at
Note that your virtualenv must be active to rerun the
make command. Note also that if you change your python code to change documentation, you must rerun
python setup.py install to pick those changes up.
This software is licensed under the Mozilla Public License 2.0