How to do regression testing

by 3 contributors:

Sometimes, our tests start to fail due to a change in Firefox. In order to investigate and fix this, we need to know what is it that changed, if it's a bug introduced or an intentional change that forces us to update our tests.

The Process of Finding a Regression Window

First, try to reproduce it manually. That gives you a hint of whether is just the test that reproduces it, maybe due to some timing issue, or if all users can experience the problem. If you can reproduce it, file a bug!

If not, lets try to see if the failure reproduces when we run the test itself only. Cause most likely we found this failure in a testrun, with the results shown in the dashboard. If it's reproducing just by running the test itself, then it would be helpful to create a minimized testcase that showcases only the issue. This has to be independent of the repository and any libraries we use. Make sure to update the bug's keyword with "testcase-wanted" or "testcase" if it's being provided, attached as plain text.

If it's not reproducing just by running the test itself, try to run the folder in which the test is, could be that a previous test caused this by not cleaning everything after it. In the worst case scenario, it's only reproducing in testruns.

When we have a fixed date when it started failing, we need to get the regression range between changes done in the last good build (where the issue does not reproduces) and the first bad build (where we can reproduce). For that, we make use of the following pushlog query:

Be aware of the repository in which you look for the pushlog, it needs to be the same branch where the failure reproduces.

The numbers you see after fromchange and tochange come from about:buildconfig, in each build (last good, first bad).

If we're not that lucky and do not know when this started, but it's within a month, we can narrow it down using thinderbox builds. In each build you try, you can see a txt file which shows which changesets have been merged in that build.

Tools

We have several tools we can use in order to get builds, install them, bisect and build some in our own:

  • Use mozdownload to download builds
  • Use the FTP server to browse and download the builds
  • Use mozregression (available via PyPI) to automatically download/install the builds and bisect to determine the causing changeset
  • Use hg bisect to automatically bisect between the last good build and first bad build - this is done over mozilla-central repository.
    • It checks out the next changeset to test
    • You need to build Firefox at that changeset  - ./mach build
    • The created build will be found under the obj_directory/dist/bin/firefox
    • Test for the regression - run the testcase or the affected mozmill test against the new build and then mark the changeset as good or bad.
    • It is possible that you hit a merge at some point, to expend that just run hg bisect --expand.
    • When it's complete you will have the failing changeset.
Recommended to take notes for what has been checked out, in case you make a mistake. 

In case you marked a changeset as good/bad by mistake, you can use hg bisect --reset to undo that. Also if a build can not be created, use hg bisect --skip to move forward to the next one.

Have you found it?

Congrats! Let the word out :) 
Save the changeset / bug responsible in bugzilla, add the keyword "regression"  and set tracking flags for the affected branches. 
You can also ask the developer for more info, he might be able to pinpoint the issue, maybe provide a fix.
If some of our tests are affected and failing constantly, we need to skip them in order to avoid further failures since we are aware of the issue. In the skip patch, please make sure you mark the Bug number.

Document Tags and Contributors

Contributors to this page: MartijnW, Sheppy, AndreeaMatei
Last updated by: MartijnW,