Mozilla Developer Network

One of the top-priority goals identified by the Gecko 1.9/Firefox 3 planning meetings was improving the test automation and testability of our code. To further this goal, I am instituting a new unit-test policy for all checkins to the toolkit module.

The Importance of Testing

Unit testing offers many advantages:

• Testing gives code authors and reviewers confidence that patches produce the correct results.
• Authoring testcases is a good impetus for developers to discover edge cases.
• Testing provides a good way to catch regressions quickly, and to make sure that no regression will be repeated twice.
• Unit tests provide working examples for how to use an API and can significantly aid documentation efforts.

Overall, integrated unit testing makes the cost and risk of any individual change smaller. It will allow the project to make the major architectural improvements of Mozilla 2 quickly and confidently.

Complete Coverage

In order for unit testing to achieve its promise, unit tests must be created along with new code. The tests must be run regularly and the results must be published where developers will notice and fix them. For Mozilla, this means that most tests of the client code should be run during the make check pass of the build system. The results of this pass are reported to the tinderbox as TUnit.

I would like to have complete unit test coverage for all toolkit code. In order to do this, I am instituting the additional checkin policies:

• All patches must include unit tests which are run during make check. Once unit tests are committed, the in-testsuite+ flag should be set on the bug.
• Unit tests must be included in patches and reviewed just like any other code.
• If running the test during make check is not possible, consult with Dave Liebrech or Robert Sayre to determine whether another test system is available.
• If a unit test framework is needed but is not yet available, the developer should write appropriate test code and commit it. A bug should be filed on the needed test framework. The in-testsuite? flag should be set on the bug until the framework has been completed and the test code is running automatically.
• Certain build-config or tooling bugs which don’t affect the actual product cannot be usefully tested. These bugs should have the in-testsuite- flag set.

In addition, I will be soliciting volunteers to go through all the old bugs in the toolkit; they will triage FIXED bugs with the in-testsuite? or in-testsuite- flag, and then help produce appropriate testcases by 1 March, 2007. This is an ambitious goal, but I believe that we can achieve it, and that it is important to get complete coverage before it becomes necessary to branch for Mozilla 2.

Robert Sayre has graciously agreed to act as toolkit unit-test czar. If you are a developer and you have questions about how to write a unit test or what unit-testing frameworks are available, please first read the documentation and then contact him (sayrer on IRC). Please don’t ask him to write the tests for you ;-). Dave Liebrech (davel on IRC) is also available to answer question and help with the design and implementation of unit-testing frameworks.

Review and Testing are Complementary

It is important to note that unit testing is not a substitute for review. Testing is important because it ensures that the code does what it is supposed to do. Review is important because it ensures that the code is well-documented, understandable by a human being, and designed to work with other code. Mozilla has traditionally been very good about requiring review; I would like to build a similar culture around unit testing. Unit testing and review are complementary activities and are both necessary for our codebase to continue to improve.

Note: please direct followup questions and discussions about this policy to the mozilla.dev.platform newsgroup (NNTP, google groups).

This entry was posted by bsmedberg on Monday, November 6th, 2006 at 7:37 pm and is filed under General.