Mozmill is not just another testing tool inside the automated testing framework provided by Mozilla. Instead it offers possibilities other test suites cannot fulfill. At the first glance it's really important to note that no dependencies to test enabled builds exist. That means there is no need to create a test enabled Firefox build on its own. But any official build whether if its a release or a nightly build will work out-of-the box. The installation of Mozmill has to be done only once. Afterward each build on the local system can be used to run the existing Mozmill tests immediately.
Mozmill Test Automation
Running functional tests with Mozmill in an automated manner is a great assistance for mozQA. In the past all those tests which are located on Litmus had to be run manually. Seeing a still increasing number of manual tests it takes longer for mozQA to run all the needed tests against release candidates or nightly builds of Firefox. The way Mozmill operates can help us to automate nearly all of those tests and let them run on all platforms and across localized builds.
To handle all the work, which has to happen to have a fully automated Mozmill test suite available, the Mozmill Test Automation project has been created. Head over to the project page and see which sub-projects we are working on and how the work is coordinated.
In the following we will give tips and tricks in using Mozmill to run our existent Mozmill tests against Firefox and how you can contribute to the project by creating new or fixing broken tests. All the information you will need to start helping out can be found below.
Installation of Mozmill
There are two different ways in using Mozmill to run tests within Firefox. First for new users the Mozmill add-on might be useful. While it makes it easy to play around with Mozmill and run the first tests it also offers an integreated development environment (IDE) to create, run, and debug tests. But there are limitations like the missing feature to run restart tests or reporting to a couchdb server. To get that functionality the command line client has to be installed instead.
The Mozmill-Test Repository
Having a central place of storage makes it always easier to distribute existent content to consumers. That's why a distributed version control system is used to manage the test repository and to give access to existent tests and our self-developed shared modules. Fortunately, this repository has already been created at http://hg.mozilla.org/qa/mozmill-tests/ and is based on Mercurial.
The Test Repository
To be able to run Mozmill tests you have to be familiar with our repository and the tools we are using. Read through this section to learn how to clone the repository, run the tests, and contributing yourself by writing or fixing tests on your own.
Before a copy of the repository can be cloned to the local disk Mercurial has to be installed by following these instructions.
With Mercurial installed the default configuration has to be prepared. All the changes should be made in the default Mercurial resource configuration file whose location can be found within this link. If the file doesn't exist on your machine you would have to create it now. Then open the config file with your preferred editor and update its contents with your own data so it will look like:
[ui] username = Your Real Name <email@example.com> merge = internal:merge (or your-merge-program) [diff] git = 1 showfunc = 1 unified = 8 [defaults] [extensions] hgext.mq = [hooks] prechangegroup.mq-no-pull = ! hg qtop > /dev/null 2>&1
As you can see a couple of entries have been added. Under the
[ui] section the
username should be set to your full name and the preferred email address. If you don't wanna use the internal merge tool specify the wanted application in the
merge line, otherwise leave
internal:merge. Within the
[diff] section the output for the diff command can be specified. It's suggested to leave the values as they stand. The next section
[extensions] enables the Mercurial Queue extension which can be used to handle a patch queue for easier management. Last but not least a hook has been added by the
[hooks] section to make sure that you don't destroy the local repository when calling "hg pull" while a patch is applied. With those changes the environment has been prepared to clone the Mozmill Test repository.
Cloning the Test Repository
The cloning process is a one time action. Once you have a copy of the repository on your machine it can be updated instead - see the next section. To clone the repository only one singly command is necessary. It will retrieve all the files from the central repository and save them to a subfolder of your choice. Therefor change into a folder of your choice before executing the
$ cd %folder% $ hg clone http://hg.mozilla.org/qa/mozmill-tests [subfolder]
Now a copy of the repository can be found under the
subfolder. If you wish to use the repository name as sub folder don't specify that parameter and a copy will be saved under
Updating the Local Copy
To always stay on the bleeding edge you have to pull the newest version of the repository regularly. With the command below all new, changed, and removed files will be updated in your local copy:
$ hg pull -u
The mozmill-tests repository contains tests for different versions of Firefox. That's necessary because ui elements or their behavior could have been changed between major versions of Firefox. With only one set of tests in place the test-run would produce test failures and make the results not reliable.
Instead of using multiple repositories for the different versions of Firefox we handle everything inside the same repository by using multiple heads. At the moment the following heads exist in the repository:
By cloning the repository the
default branch is selected automatically. As long as the tests will be run against Firefox 3.6.x there will be no problems. But once you want to run the tests against an older version, the head has to be switched. To check which branches exist run the command below and you will get a list with the revision id of the latest check-in.
$ hg branches default 289:86effcb2ddd9 mozilla1.9.1 288:0b20105de12c
If you do not know which branch is actually selected run:
$ hg branch default
Given the output the default branch is currently selected and the tests will work with all versions of Firefox 3.6.x. If another branch is needed because tests have to be run against Firefox 3.5.x, the following command switches to the mozilla1.9.1 branch:
$ hg up -C mozilla1.9.1 84 files updated, 0 files merged, 1 files removed, 0 files unresolved
The repository and all its test will be updated to the latest version of tests in that branch.
Running Mozmill Tests
To get familiar with Mozmill test scripts you can take a look at the exisiting Firefox tests from the Mozmill-Test repository. If you haven't pulled them on your local disk please do that now. See the chapter above for instructions.
Depending on the method you have used to install Mozmill there are two different ways to execute Mozmill tests. You can run them from within the Mozmill IDE which is part of the Firefox add-on or by executing the command line application. The latter offers additional features like restart capabilities for Mozmill tests.
If you wanna use the Mozmill IDE to run tests please see the introductions in how to use it. Otherwise the proposed way is to use the command line application and its usage will be described below
The Command Line Client
By using the command line client Mozmill tests can be executed but you can also open Firefox in the developer mode which let you debug web pages or Firefox itself while also giving access to the Mozmill IDE. To run tests, execute the client with one of the options given below. A fresh profile will be automatically created so the test always runs in a clean environment. But keep in mind that if you want to run a bunch of tests inside a folder all those tests will be executed in the same profile. Beneath those normal tests you will also be able to run restart tests like what is needed for extension installations.
Once you have installed Mozmill via setuptools you can run the
mozmill client with the
--help option to get a list of available options:
$ mozmill --help usage: mozmill [options] Options: -h, --help show this help message and exit -b BINARY, --binary=BINARY Binary path. -w PLUGINS, --plugins=PLUGINS Plugin paths to install. -n, --no-new-profile Do not create new profile. -l LOGFILE, --logfile=LOGFILE Log all events to file. --report=REPORT Report the results. Requires url to results server. -t TEST, --test=TEST Run test file or directory. --showall Show all test output. -D, --debug Install debugging plugins. -d DEFAULT_PROFILE, --default-profile=DEFAULT_PROFILE Default profile path. --show-errors Print logger errors to the console. -u, --usecode Use code module instead of iPython -s, --shell Start a Python shell -p PROFILE, --profile=PROFILE Profile path. -P PORT, --port=PORT TCP port to run jsbridge on.
From the list of options to customize the Mozmill test-run you will normally use three options: The most important option is
--test which specifies a single test file or a test folder and its sub folders where the tests resist. The
--binary option is useful to run the tests against a specified version of Firefox instead of letting Mozmill find the systems default Firefox browser. Finally
--show-errors can be used to get a more comprehensive error output in the shell window. Below you can find some examples specific to our mozmill-test repository for Firefox.
Run normal Mozmill Tests
To run our normal Mozmill tests the
mozmill command has to be used.
$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js
Starts the default Firefox application, executes the given test, and closes Firefox afterward.
$ mozmill -t firefox/testPreferences/
Starts the default Firefox application, executes all the tests in the given folder and its sub folders, and closes Firefox afterward.
$ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "c:\firefox 3.5\firefox.exe" (Windows) $ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "/usr/bin/firefox" (Linux) $ mozmill -t firefox/testPreferences/testRestoreHomepageToDefault.js -b "/Applications/Firefox.app" (Mac OS X)
Starts the specified version of Firefox (as shown Windows, Linux, or OS X), executes the given test, and closes the browser afterward.
--binaryoption the full path to the executable has to be specified on Windows and Linux while on OS X the application is used.
Run Mozmill Restart Tests
Restart tests can be executed by using the
mozmill-restart command. It allows you to run tests like installing an extension which needs a restart to finish. For restart tests you will always specify a test folder for the
--test option. It will run all the test files in that folder in an alphabetical order.
$ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/
Starts the systems default Firefox application, runs all the tests under the given folder by restarting Firefox in-between each test, and finally closes Firefox. The same profile is used for all test files inside this folder.
$ mozmill-restart -t firefox/restartTests/
Starts the systems default Firefox application, runs the restart tests for all sub folders, and finally closes Firefox. The same profile is only used for one sub folder. It will not be shared between the different sub folders.
$ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/ -b "c:\firefox 3.5\firefox.exe" (Windows) $ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/ -b "/usr/bin/firefox" (Linux) $ mozmill-restart -t firefox/restartTests/testExtensionInstallUninstall/ -b "/Applications/Firefox.app" (Mac OS X)
Starts the specified version of Firefox, runs all the tests in the given folder by restarting Firefox in-between each test, and closes the browser afterward.
--binaryoption the full path to the executable has to be specified on Windows and Linux while on OS X the application is used.
Writing Mozmill Tests
With the knowledge how to run Mozmill tests you can help in writing new tests or by fixing existing ones. It's not a complicated task but you have to obey some simple rules so we can guarantee long living and understandable tests for everyone.
How to Start
To make it easier for you to create your first Mozmill tests we have prepared a couple of template files for your usage. They will help you to get familiar with the license block, needed test functions, shared modules, and they way how tests have to be correctly written syntax wise. You can find these files in your local version of the test repository or online.
Take care about the following items when using the templates:
- Please update the name and the email address in the license block
- Update the Litmus meta data at the end of the file to show the correlation to the corresponding Litmus test. If multiple Litmus tests are covered in a test file add separate meta entries for each test function.
- Use a meaningful name for your test function which indicates the overall target of the test.
Additional to the initial steps to the test files there are some other rules you should take care of. It will help us to better understand the tests:
- Use comments wisely. Group lines of code belonging together and give those a combined comment. See a shared module as example.
- Use the Litmus testcase description for naming your test file and save it into a subfolder of the firefox directory which has the same name as the Litmus subgroup. If you are working on a restart test the file naming convention says to use testX.js where X is a number. Place all test files into its own folder under firefox/restartTests.
If that was not enough information you should have a look at the already existing tests in the Mozmill Test repository.
Tips and Tricks
Sometimes you will run into trouble while creating Mozmill tests. It's our attitude to help you in solving those problems and to simplify your test creation process.
- Make sure to get familiar with the functionality provided by Mozmill and all of our existent Shared Modules which ease the test creation process.
- Use the Inspector or Recorder to create the skeleton of your test. You have to add additional steps like calls to sleep functions or element checks before the test can be run.
- If you are using controller.open() to load a web page a controller.waitForPageLoad() has to be used right afterward. A controller.sleep() call is not sufficient.
- Use the controller's menu API to reach commands which are only available via the main menu. A list of existing id's for menu items can be found in the browser-menubar.inc file. Due to our localization efforts please always use the id of menu items.
- If your test needs exactly one tab open use "TabbedBrowsingAPI.closeAllTabs(controller);" inside the setupModule function.
- If you modify preferences or other global data make sure to reset those values inside the teardownModule function. That will clean-up the environment for the next Mozmill tests.
- Avoid using any hardcoded strings for the elementslib Lookup function. Those will break Mozmill tests for localized builds. After using the inspector you have to manually remove those attributes (e.g. label or accesskey) from the element string (see the next bullet).
- If an element can only be referenced by the elementslib Lookup function please try to remove as much as possible attributes from each hierarchy. That will make the test more readable and can avoid failed lookups when the code in Firefox changes.
The Review Process
Before your test can be checked into the mozmill-test repository you have to pass the review process. The reviewer has to learn about the test and check if everything is done correctly. In order to make the review as easy as possible, make sure to check your test script abides by the guidelines given above. Besides checking the syntax and code style of the test, make sure the test runs with the command line client before requesting a review. If questions arise feel free to ask in #qa or the mozmill-dev mailing list at any time.
Simplified Patch Creation
The easiest way to create a patch is by using the '
hg diff' command bounded by two other commands. With '
hg add' you advice Mercurial to start tracking your test file. It's needed to see your test content in the diff output. Once the patch has been created you can use '
hg rm' to safely remove the test from the tracking list. That will guarantee that no conflicts will happen when you pull a new version to your local copy of the repository.
Imagine you have created a test called "testZoomSettings.js" which is saved under "firefox/testLayout/" and you want to create a patch called "patch_file":
$ hg add firefox/testLayout/testZoomSettings.js $ hg diff >patch_file $ hg rm -f firefox/testLayout/testZoomSettings.js
After you ran those commands you will find the file
patch_file in the current folder which can be uploaded as attachment to the bug report.
Advanced Patch Creation
As you can imagine it will be hard to track all the files when you are working on several tests in parallel. Because all those files will lingering around in your working copy. To prevent that and to gain the overview you can use the Mercurial Queue extension.
In the example below you can see how it will work starting with a new test named
$ hg qnew zoomsettings (Add a new named patch to the queue of patches) $ vi firefox/testLayout/testZoomSetting.js (Create your test and and apply the template structure) $ hg add firefox/testLayout/testZoomSettings.js (Start tracking the test file) $ hg diff (Create a diff output of the current state) $ hg qrefresh (Update the patch by accepting all changes) $ vi firefox/testLayout/testZoomSetting.js (Continue to update your test) $ hg diff (Create a diff against the last version of your patch) $ hg qdiff (Create a complete diff against the current version of the repository) $ hg qrefresh (Refresh the patch with the latest changes) $ hg qpop (Pop the patch from the stack) $ hg qpush (Push the patch back to the stack) $ hg export tip >patch_file (Create a patch based on the current state)
When using the advanced way of creating a patch all existing patches are located under .hg/patches. Before you ask for a review you should check the patch if it is valid. You can use the online review tool. There should be listed only comments which claims that lines are too long.
Reviews are managed in Bugzilla. So a new bug report for this particular test has to be created (see bug 479720 as example). The new bug should be filed in the product and component your test is executed against (e.g. if you write a security test use Firefox/Security). Also add the Litmus testcase id's for all branches in the first comment of the bug and use the
[mozmill] prefix in the summary. Finally your test has to be attached to the bug. Now you can request review from Aakash Desai or Henrik Skupin.
Testing extensions with Mozmill
Beside using Mozmill to test the application itself, it is also possible to test any kind of extension. The initial steps which have to be done to be able to run a first test against an extension will be described below.
Creating a test
Creating a test for an extension is not much different from what you have learned so far. Check the above sections how to start and implement a test for Mozmill.
Running a test
Once the test file has been created and a couple of tests have been added, this test file can be run with Mozmill inside the desired application. Therefor the extension has to be installed first. When using the Mozmill add-on, the extension would have to be installed manually. But with the command line client everything can be automated:
- Download the version of the extension you want to test to the local drive, e.g. to "
- Run the test with Mozmill and supply the extension with the
$ mozmill -b"c:\firefox 3.5\firefox.exe" -t testExtension.js -w "c:\mozmill\foobar.xpi"
Using the repository
You can store all the tests locally on your hard-drive or you can also make them publicly available for everyone. If you don't have any problems with our MPL license your tests can be also included in our mozmill-test repository.
In order to get tests checked in, make sure that the following conditions are fulfilled:
- Create a patch for all the tests which should be checked-in. See the above coding guidelines and how the review process works.
- Your tests should be located under a sub-folder named like the extension id inside the
- All tests have to pass when they get run with Mozmill in the appropriate Firefox release. Please see above how to work with branches.
If questions remain don't hesitate to contact Henrik Skupin.