Mozmill is not just another testing tool inside the automated testing framework provided by Mozilla. Instead it offers possibilities other test suites cannot fulfill. At the first glance it's really important to note that no dependencies to test enabled builds exist. That means there is no need to create a test enabled Firefox build on its own. But any official build whether if its a release or a nightly build will work out-of-the box. The installation of Mozmill has to be done only once. Afterward each build on the local system can be used to run the existing Mozmill tests immediately.
Mozmill Test Automation
Running functional tests with Mozmill in an automated manner is a great assistance for mozQA. In the past all those tests which are located on Litmus had to be run manually. Seeing a still increasing number of manual tests it takes longer for mozQA to run all the needed tests against release candidates or nightly builds of Firefox. The way Mozmill operates can help us to automate nearly all of those tests and let them run on all platforms and across localized builds.
To handle all the work, which has to happen to have a fully automated Mozmill test suite available, the Mozmill Test Automation project has been created. Head over to the project page and see which sub-projects we are working on and how the work is coordinated.
In the following we will give tips and tricks in using Mozmill to run our existent Mozmill tests against Firefox and how you can contribute to the project by creating new or fixing broken tests. All the information you will need to start helping out can be found below.
Installation of Mozmill
There are two different ways in using Mozmill to run tests within Firefox. First for new users the Mozmill add-on might be useful. While it makes it easy to play around with Mozmill and run the first tests it also offers an integreated development environment (IDE) to create, run, and debug tests. But there are limitations like the missing feature to run restart tests or reporting to a couchdb server. To get that functionality the command line client has to be installed instead.
The Mozmill-Test Repository
Having a central place of storage makes it always easier to distribute existent content to consumers. That's why a distributed version control system is used to manage the test repository and to give access to existent tests and our self-developed shared modules. Fortunately, this repository has already been created at http://hg.mozilla.org/qa/mozmill-tests/ and is based on Mercurial.
Working with the Test Repository
To be able to run Mozmill tests you have to be familiar with our repository and the tools we are using. Read through this section to learn how to clone the repository, run the tests, and contributing yourself by writing or fixing tests on your own.
Before a copy of the repository can be cloned to the local disk Mercurial has to be installed by following these instructions.
With Mercurial installed the default configuration has to be prepared. All the changes should be made in the default Mercurial resource configuration file whose location can be found within this link. If the file doesn't exist on your machine you would have to create it now. Then open the config file with your preferred editor and update its contents with your own data so it will look like:
[ui] username = Your Real Name <firstname.lastname@example.org> merge = internal:merge (or your-merge-program) [diff] git = 1 showfunc = 1 unified = 8 [defaults] [extensions] hgext.mq = [hooks] prechangegroup.mq-no-pull = ! hg qtop > /dev/null 2>&1
As you can see a couple of entries have been added. Under the
[ui] section the
username should be set to your full name and the preferred email address. If you don't wanna use the internal merge tool specify the wanted application in the
merge line, otherwise leave
internal:merge. Within the
[diff] section the output for the diff command can be specified. It's suggested to leave the values as they stand. The next section
[extensions] enables the Mercurial Queue extension which can be used to handle a patch queue for easier management. Last but not least a hook has been added by the
[hooks] section to make sure that you don't destroy the local repository when calling "hg pull" while a patch is applied. With those changes the environment has been prepared to clone the Mozmill Test repository.
Cloning the Test Repository
The cloning process is a one time action. Once you have a copy of the repository on your machine it can be updated instead - see the next section. To clone the repository only one singly command is necessary. It will retrieve all the files from the central repository and save them to a subfolder of your choice. Therefor change into a folder of your choice before executing the
$ cd %folder% $ hg clone http://hg.mozilla.org/qa/mozmill-tests [subfolder]
Now a copy of the repository can be found under the
subfolder. If you wish to use the repository name as sub folder don't specify that parameter and a copy will be saved under
Updating the Local Copy
To always stay on the bleeding edge you have to pull the newest version of the repository regularly. With the command below all new, changed, and removed files will be updated in your local copy:
$ hg pull -u
The mozmill-tests repository contains tests for different versions of Firefox. That's necessary because ui elements or their behavior could have been changed between major versions of Firefox. With only one set of tests in place the test-run would produce test failures and make the results not reliable.
Instead of using multiple repositories for the different versions of Firefox we handle everything inside the same repository by using multiple heads. At the moment the following heads exist in the repository:
By cloning the repository the
default branch is selected automatically. As long as the tests will be run against Firefox 3.6.x there will be no problems. But once you want to run the tests against an older version, the head has to be switched. To check which branches exist run the command below and you will get a list with the revision id of the latest check-in.
$ hg branches default 289:86effcb2ddd9 mozilla1.9.1 288:0b20105de12c
If you do not know which branch is actually selected run:
$ hg branch default
Given the output the default branch is currently selected and the tests will work with all versions of Firefox 3.6.x. If another branch is needed because tests have to be run against Firefox 3.5.x, the following command switches to the mozilla1.9.1 branch:
$ hg up -C mozilla1.9.1 84 files updated, 0 files merged, 1 files removed, 0 files unresolved
The repository and all its test will be updated to the latest version of tests in that branch.