Web UI: https://screenshots.mattn.ca/compare/
browser/tools/mozscreenshots/ (requires full `./mach build` to rebuild)
Web, fetch and compare source: https://github.com/mnoorenberghe/mozscreenshots
Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=mozscreenshots
Try syntax example:
try: -b o -p linux,linux64,macosx64,win32,win64 -u mochitest-browser-screenshots[Ubuntu,10.10,Windows XP,Windows 7,Windows 8] -t none --setenv MOZSCREENSHOTS_SETS=TabsInTitlebar,Tabs,WindowSize,Toolbars,LightweightThemes,Preferences
Mach try example (additional quotes required):
./mach try -b o -p linux,linux64,macosx64,win32,win64 -u mochitest-browser-screenshots["Ubuntu","10.10","Windows XP","Windows 7","Windows 8"] -t none --setenv MOZSCREENSHOTS_SETS=TabsInTitlebar,Tabs,WindowSize,Toolbars,LightweightThemes,Preferences
Simple steps to using mozscreenshots
- Apply your changes locally to your source control repository.
- With one of the syntax examples above, either:
- Use "mach try", or
- create a new changeset on top of your changes and push your changes to tryserver.
- Wait a while, then visit the treeherder results page. On the page, look for the Tiers button on the second row of the menu. Select "Tier 3" and then unselect "Tier 1" and "Tier 2". Then check the box next to the "Tiers" button to enable showing excluded builds. Make sure that the "ss" jobs are green.
- Visit the web UI: https://screenshots.mattn.ca/compare/
- Write "mozilla-central" in as the project for the base. Put focus in the revision field and press down to get the form autocomplete to show. Select one of the recent Nightly build dates to use as a base revision.
- Write "try" in as the project for the new revision. Paste in the longform of the try revision changeset hash.
- Click on Compare Revisions
- An indeterminate progress will appear while results are loading. You can uncheck "Hide similar" to show rows where there were no differences. This helps to confirm that the diffs are taking place.
You can request screenshots be captured on a Try push for UI review or comparison to a know-good base by adding requesting the “mochitest-browser-screenshots” test job and specifying what you would like captured by setting the
MOZSCREENSHOTS_SETS environment variable with a comma-separated list of configurations like so:
Screenshots will only be captured for opt builds. Note that the job is currently Tier 3 and “excluded” on TreeHerder so you will need to toggle both of those filters to see the jobs there with the symbol: M[Tier-3] (ss). Try pushes, unlike Nightlies, won’t capture any images by default if
MOZSCREENSHOTS_SETS isn’t specified in order to avoid capturing images when developers request all mochitest runs but don’t really want the screenshots.
mach mochitest --subsuite screenshots --setenv MOZSCREENSHOTS_SETS=DevEdition,TabsInTitlebar,Tabs,WindowSize,Toolbars
The screenshots can be found in the following directories:
MOZ_UPLOAD_DIRif the environment variables is defined
/%TmpD%/mozscreenshots/(a temporary directory which is output at the before capturing begins)
On mozilla-central Nightly builds (with an update channel of "nightly"), the following configurations are run:
- TabsInTitlebar,Tabs,WindowSize,Toolbars,LightweightThemes –
Fetching from automation
If you simply want to compare images from the Web UI, you can skip this section. If you want to download the images for analysis or review locally, you can use
The simplest way to download all captured images for a push is to use the fetch_screenshots command (which I hope to add to mach):
pip install -U mozscreenshots fetch_screenshots -r <my_try_revision>
fetch_screenshots --nightly 2016-02-10
The captured screenshots also show up in the Job details as artifacts on TreeHerder, if you want to manually inspect them.
The simplest way to compare images is via the web UI at http://screenshots.mattn.ca/compare/.
When you visit the above URL, you will have to fill in a base (e.g. the Nightly revision from about:buildconfig) and new revision (e.g. a try push with some patches to review).
- If you are comparing against a recent Nightly, just enter "mozilla-central" (no quotes) in to the Project field, and then put focus in the related Revision field and press the Down key on your keyboard. In the form autocomplete you will see references to recent Nightly builds, which when selected will populate the field with the Nightly build's revision changeset.
In the background the images are fetched from automation via
fetch_screenshots and then compared using
compare_screenshots with the output displayed on the page. The first comparison for a pair of revisions can take several minutes as around one thousand (5 platforms x 2 revisions x 100 screenshots) images need to be downloaded and compared for the default set of screenshots. Subsequent comparisons for the same revision are much faster as the results are cached.
compare_screenshots to compare image files or directories (recursively) using ImageMagick. System UI (e.g. the clock and taskbar) is cropped out of the images when necessary so they aren't included in image comparisons and generate false positives.
pip install -U mozscreenshots compare_screenshots mozilla-central/08138045c38c/ try/5f6ca9194dd9/
Why don't I see the job/results on TreeHerder?
The mochitest-browser-screenshots jobs are currently Tier 3 and marked as Excluded so you will need to toggle the filters to make the jobs visible.
--setenv MOZSCREENSHOTS_SETS=… isn't specified, no screenshots will be captured on Try server.
How is this different than reftests?
mochitest-browser-screenshots is unlike reftests in that no comparison to a reference image is done on test machines, only capturing. Eventually we may move to a model more like reftests if this is successful but that would also imply capturing on every integration push in order to identify the regressing push which uses more resources. We’ll also see how often the reference images would have needed an update using the current workflow to inform the decision.
How do I capture screenshots of something not supported by mozscreenshots?
Simply add a JSM at
browser/tools/mozscreenshots/mozscreenshots/extension/configurations/ and then select it with the base name of the JSM.