Browser screenshots

Web UI: screenshots.mattn.ca/compare/
Capture source: browser/tools/mozscreenshots/ (may require full `./mach build` to rebuild certain files)
IRC channel: #mozscreenshots
Alert mailing list: mozilla.dev.ui-alerts
Code and issues for web, fetch and compare: github.com/mnoorenberghe/mozscreenshots
Component for mozscreenshots harness bugs: Testing::mozscreenshots (file a bug)
Meta Bug for issues in configurations:  bugzil.la/mozscreenshots

Try syntax example:
try: -b o -p linux,linux64,macosx64,win32,win64 -u browser-screenshots-e10s[Ubuntu,10.10,Windows 7,Windows 10] -t none
Mach try example (additional quotes required):
mach try fuzzy -q "browser-screenshots"

Simple steps to using mozscreenshots on Try

  1. Apply your changes locally to your source control repository.
  2. With one of the syntax examples above, either:
    1. Use mach try, or
    2. create a new changeset on top of your changes and push your changes to tryserver.
  3. Wait a while, then visit the treeherder results page. On the page, look for the Tiers button on the second row of the menu. Select "Tier 3" and in the search box type "screenshot" and hit enter. Make sure that the "ss" jobs are green.
  4. Visit the web UI: https://screenshots.mattn.ca/compare/
    1. Write "mozilla-central" in as the project for the base. Put focus in the revision field and press down to get the form autocomplete to show. Select one of the recent mozilla-central build dates to use as a base revision.
    2. Write "try" in as the project for the new revision. Paste in the longform of the try revision changeset hash.
    3. Click on Compare Revisions
      • An indeterminate progress will appear while results are loading. You can uncheck "Hide similar" to show rows where there were no differences. This helps to confirm that the diffs are taking place.

Capturing

mozilla-central

On mozilla-central opt builds there are many configurations captured and comparison results are reported to the mozilla.dev.ui-alerts list.

Try pushes

You can request screenshots be captured on a Try push for UI review or comparison to a know-good base by requesting the “browser-screenshots-e10s” test job on opt builds. By default, Try pushes capture the same images as mozilla-central builds but you can override what you would like captured by setting the MOZSCREENSHOTS_SETS environment variable with a comma-separated list of configurations like so in legacy try syntax:

--setenv MOZSCREENSHOTS_SETS=DevEdition,TabsInTitlebar,Tabs,WindowSize,Toolbars

(Use --env (no "set") for mach try fuzzy)
Note that the job is currently Tier 3 on TreeHerder so you will need to toggle it on to see the jobs there with the symbol: tc-M-e10s(ss). The capture job won't run for try syntax of "-u all" or "-u mochitests" in order to save resources by not capturing images unless specifically requested. When using legacy try syntax, you must also specify the test platforms in square brackets after the job name.

bug 1416538 causes the branding for artifact builds to differ from non-artifact builds and cause false-positives in comparisons. Forcing a non-artifact try push with --no-artifact will avoid the issue.

Other trees

You can run mozscreenshots on any other mozilla-central-based tree using the "Add New Jobs" menu in TreeHerder for a push. By default it will capture the same images as a mozilla-central build but in theory you can use the experimental "Custom Push Action" menu to override the MOZSCREENSHOTS_SETS environment variable.

mach

mach mochitest --subsuite screenshots --setenv MOZSCREENSHOTS_SETS=DevEdition,TabsInTitlebar,Tabs,WindowSize,Toolbars

The screenshots can be found in the following directories:

  • In MOZ_UPLOAD_DIR if the environment variables is defined
  • /%TmpD%/mozscreenshots/ (a temporary directory which is output at the before capturing begins)

 

Fetching from automation

If you simply want to compare images from the Web UI, you can skip this section. If you want to download the images for analysis or review locally, you can use fetch_screenshots.

The simplest way to download all captured images for a push is to use the fetch_screenshots[8] command (which I hope to add to mach):

pip install -U mozscreenshots
fetch_screenshots -r <my_try_revision>

or

fetch_screenshots --nightly 2016-02-10

The captured screenshots also show up in the Job details as artifacts on TreeHerder, if you want to manually inspect them.

Comparing

Web UI

The simplest way to compare images is via the web UI at http://screenshots.mattn.ca/compare/.

When you visit the above URL, you will have to fill in a base (e.g. the Nightly revision from about:buildconfig) and new revision (e.g. a try push with some patches to review).

  • If you are comparing against a recent mozilla-centra/Nightly build, just enter "mozilla-central" (no quotes) in to the Project field, and then put focus in the related Revision field and press the Down key on your keyboard. In the form autocomplete you will see references to recent mozilla-central builds, which when selected will populate the field with the m-c build's revision changeset.

In the background the images are fetched from automation via fetch_screenshots and then compared using compare_screenshots with the output displayed on the page. The first comparison for a pair of revisions can take several minutes as around one thousand (5 platforms × 2 revisions × 100 screenshots) images need to be downloaded and compared for the default set of screenshots. Subsequent comparisons for the same revision are much faster as the results are cached.

Command line

Use compare_screenshots to compare image files or directories (recursively) using ImageMagick. System UI (e.g. the clock and taskbar) is cropped out of the images when necessary so they aren't included in image comparisons and generate false positives.

pip install -U mozscreenshots
compare_screenshots  mozilla-central/08138045c38c/ try/5f6ca9194dd9/

FAQ

Why don't I see the job/results on TreeHerder?

The browser-screenshots-e10s jobs are currently Tier 3 so you will need to toggle the filters to make the jobs visible.

If --setenv MOZSCREENSHOTS_SETS=… isn't specified, no screenshots will be captured on Try server.

How is this different than reftests?

browser-screenshots-e10s is unlike reftests in that no comparison to a reference image is done on test machines, only capturing. Eventually we may move to a model more like reftests if this is successful but that would also imply capturing on every integration push in order to identify the regressing push which uses more resources. We’ll also see how often the reference images would have needed an update using the current workflow to inform the decision.

How do I capture screenshots of something not supported by mozscreenshots?

Simply add a JSM at browser/tools/mozscreenshots/mozscreenshots/extension/configurations/ and then select it with the base name of the JSM.

Document Tags and Contributors

Contributors to this page: fitojb, mnoorenberghe, JaredWein, ktbee, P.A., johannh
Last updated by: fitojb,