Join MDN and developers like you at Mozilla's View Source conference, 12-14 September in Berlin, Germany. Learn more at https://viewsourceconf.org

Mozmill commit policy

The following details our policy for committing patches to the mozmill-tests and litmus-data repositories. If you have any questions or need help email Anthony Hughes or ping ashughes on irc.mozilla.org/#automation.

What is a committer?

A committer is someone who physically checks code into the repository. This is a privilege which comes with a lot of responsibility and is typically only granted to those who have proven themselves over an extended period of contribution.

How do I become a committer?

When the team feels you are ready to become a committer, you will need to go through the application process. Mozilla has a standard Committer Agreement with everyone who wishes to check code into any repository. If you want to be able to land patches to the mozmill-tests and litmus-data repositories, get in touch with Anthony Hughes.

Privileges

Once granted commit access you will be granted the following privileges:

  • Landing patches to skip/disable a failing test (patch must be approved by one peer)
  • Landing patches to fix a failing test (patch must be approved by one peer and one super-reviewer)
  • Landing patches to add a new test (patch must be approved by one peer and one super-reviewer)
  • Landing patches to litmus-data (patch must be approved by one peer and one super-reviewer)
  • Landing patches to endurance tests (patch must be approved by one peer, one super-reviewer, and the module owner)
  • Landing patches to shared modules (patch must be approved by one peer, one super-reviewer, and the module owner)

Responsibilities

Once granted commit access you will be granted the following responsibilities:

  • Landing your own patches after they have passed review (ideally within 24 hours of r+)
  • Checking the live test results the following day to see if your patch introduced a regression
  • Promptly backing out your patch when a regression is introduced
  • Landing patches for those without commit access after they have passed review
Note: If you land a bad patch, it is your responsibility to back it out. If you write a bad patch, it is your responsibility to fix it.

Life-cycle of a patch

Patches for Endurance Tests

  • Land the patch on the default branch
  • Update the bug with the following changes
    • [landed] added to the patch summary
    • comment with the URL to the changeset
    • status set to RESOLVED FIXED
  • The next day, verify the test has not caused regressions by checking the Endurance Dashboard for failures
    • if no failures are found, set the bug status to VERIFIED FIXED
    • if the test fails, the bug status is set to REOPENED and the patch is backed out

Patches for Functional Tests

  • Land the patch on the default branch
  • Update the bug with the following changes
    • [landed:default] added to the patch summary
    • comment with the URL to the changeset
    • status set to RESOLVED FIXED
  • The next day, verify the test has not caused regressions by checking the Functional Dashboard for failures
  • If no failures are found, land the patch on the mozilla-aurora, mozilla-beta, and mozilla-release branches, and update the bug with the following changes:
    • change [landed:default] to [landed] in the patch summary
    • comment with the URLs to the changesets for each of the branches
    • change the status to VERIFIED FIXED
  • The next day, verify the test has not caused regressions by checking the Functional Dashboard for failures
    • if the test fails, the bug status is set to REOPENED and the patch is backed out
NOTE: Only land your patch on branches where the feature exists. For example, if your test is for a feature which is only active on Nightly, land the patch on default; if your test is for a feature which is active on Beta, land your patch on default, mozilla-aurora, and mozilla-beta; and so on.
NOTE: Be sure to update the corresponding Litmus test with a link the the Mozmill test. Also, clone the Litmus test to the Aurora Mozmill Tests subgroup.

Patches for Test Failures

  • Land the patch on the affected branch
  • Update the bug with the following changes
    • [landed] added to the patch summary
    • comment with the URL to the changeset
    • status set to RESOLVED FIXED
  • The next day, verify the test has not caused regressions by checking the Functional Dashboard for failures
  • If the test is still failing, set the bug status to REOPENED and back out the patch
  • Otherwise, change the status of the bug to VERIFIED FIXED
NOTE: Any failures which only affect the mozilla-1.9.2 branch will be disabled, the bug resolved WONTFIX, otherwise follow this process:
     1. Disable the test
     2. Investigate if the bug is a failure of the test or Firefox
     3. If it's a problem with the test, the test will remain disabled until it can be fixed
     4. If it's a problem with Firefox, file a Firefox bug, re-enable the test and close the test failure bug
NOTE: Any disabled test should also be disabled, any fixed test re-enabled, in Mozmill Tests subgroup in Litmus

Litmus

After you have landed a new test, or disabled a failing one, you need to update Litmus.

  • If the test has been disabled
    • find the Mozmill Tests subgroup for each branch in Litmus
    • remove the test from the subgroup
    • amend the Mozmill Test block to include the word DISABLED and strike-through the link to the test file
    • Remember to revert this change when the test is being re-enabled
  • If the test is new
    • find the Litmus testcase corresponding to your Mozmill test
    • add a comment to the test stating it is covered by Mozmill and provide a link to the file
    • add the Litmus test to the Mozmill Tests subgroup for that branch
    • This needs to be done for each and every branch the test has landed

Nomenclature for patch landing

When you land a patch you should do the following in Bugzilla:

  • Edit Details for the attached patch
    • If the patch is landed on all branches, add [landed] to the attachment summary
    • If the patch is landed on a single branch, add [landed:branch_name] to the attachment summary
  • Add a comment to the bug indicating where the patch was landed and provide a link to the changeset

Example:

Comment on attachment 588432 [details] [diff] [review]
test v7 [landed]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/829eb7f3fc87 (mozilla-aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/49cea842da76 (mozilla-beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/819e3aa37289 (mozilla-release)

Non-responsibilities

The following are not your responsibility as a committer and should never be done unless given explicit permission. These will be take care of by the module owners.

  • Landing patches for new tests on the mozilla-1.9.2 branch
  • Landing patches on the mozilla-esrN branch
  • Merging branches

Document Tags and Contributors

 Contributors to this page: Sheppy, Ashughes
 Last updated by: Sheppy,