Mozmill tests

この文書は翻訳中です。他国語のままの部分などがあるのはその為です。
是非お気軽に MDN に登録して翻訳に参加し、私たちの手助けをして下さい!

Mozmill is not just another testing tool inside the automated testing framework provided by Mozilla. Instead it offers possibilities other test suites cannot fulfill. At first glance, it's really important to note that no dependencies to test enabled builds exist. That means there is no need to create a special "test enabled" Firefox build before using Mozmill; instead, any official build including releases and nightly builds will work out-of-the box. The installation of Mozmill need only be done once. After that, each build on the local system can be used to run the existing Mozmill tests immediately.

Mozmill tests are written in JavaScript and get executed in the scope of the browser window, which enables them to have access to any part of the UI and also to all available XPCOM components. Using Mozmill's command line client also offers the ability to run tests which require a restart of the application.

Mozmill test automation

Running functional tests with Mozmill in an automated manner is very helpful for mozQA. In the past all the tests had to be run manually. Seeing a still increasing number of manual tests it takes longer for mozQA to run all the needed tests against release candidates or nightly builds of Firefox. The way Mozmill operates can help us to automate nearly all of those tests and let them run on all platforms and across localized builds.

To handle all the work that needs to be done in order to have a fully automated Mozmill test suite available, the Mozmill Test Automation project has been created. Head over to the project page and see which sub-projects we are working on and how the work is coordinated.

In the following we will give tips and tricks on using Mozmill to run our existing Mozmill tests against Firefox and how you can contribute to the project by creating new or fixing broken tests. All the information you will need to start helping out can be found below.

Installing Mozmill

You can find detailed step-by-step installation instructions on the Mozmill page. Just make sure that you install Mozmill as a command line application via PyPI.

The Mozmill-Test repository

Having a central place of storage makes it always easier to distribute existent content to consumers. That's why a distributed version control system is used to manage the test repository and to give access to existent tests and our self-developed shared modules. Fortunately, this repository has already been created at http://hg.mozilla.org/qa/mozmill-tests/ and is based on Mercurial.

The test repository

To be able to run Mozmill tests, you have to be familiar with our repository and the tools we're using. Read through this section to learn how to clone the repository, run the tests, and contribute by writing or fixing tests.

Mercurial Installation

Before a copy of the repository can be cloned to the local disk, Mercurial has to be installed by following these instructions.

Configuring Mercurial

With Mercurial installed, the default configuration has to be prepared. All the changes should be made in the default Mercurial resource configuration file. If the file doesn't exist on your machine, you should create it; then open the file with your preferred editor and update its contents so it includes the configuration information below:

[ui]
username = Your Real Name <user@example.com>
merge = internal:merge (or your-merge-program)

[diff]
git = 1
showfunc = 1
unified = 8

[defaults]
qnew = -U

[extensions]
hgext.color =
hgext.mq =
hgext.transplant =

[hooks]
pretxncommit.whitespace = hg export tip | (! egrep '^\+(.*[ ]*|[\t]*)$')
prechangegroup.mq-no-pull = ! hg qtop > /dev/null 2>&1

As you can see, a couple of entries have been added. Under the [ui] section the username should be set to your full name and the preferred email address. If you don't want to use the internal merge tool, you can specify your preferred application in the merge line; otherwise you can leave it set to internal:merge. Within the [diff] section, the output for the diff command can be specified. It's suggested to leave the values as they stand. The next section [extensions] enables the Mercurial Queue and Transplant extension which can be used to handle a patch queue for easier management. Last but not least, hooks have been added in the [hooks] section to make sure that no trailing white-spaces are introduced and that you don't destroy the local repository when calling "hg pull" while a patch is applied. With those changes the environment has been prepared to clone the Mozmill test repository.

Cloning the test repository

The cloning process is a one time action. Once you have a copy of the repository on your machine it can be updated instead; see the next section. Cloning the repository only requires one command, which will retrieve all the files from the central repository and save them to a subfolder of your choice. Change into a folder of your choice before executing the hg clone command:

$ cd %folder%
$ hg clone http://hg.mozilla.org/qa/mozmill-tests [subfolder]

Now a copy of the repository can be found under the specified subfolder. If you wish to use the repository name as the name of the subfolder, don't specify that parameter and a copy will be saved under mozmill-tests.

Updating the local copy

To always stay on the bleeding edge, you have to pull the newest version of the repository regularly. With the command below, all new, changed, and removed files will be updated in your local copy (run this in the specified subfolder of the cloned repository, where an .hg file is located):

$ hg pull -u
Note: Before you run any of the Mozmill tests in Firefox make sure you have the latest revision checked out.

Handling branches

The mozmill-tests repository contains tests for different versions of Firefox. That's necessary because UI elements or their behavior could have been changed between major versions. With only one set of tests and modules in place, the test-run would produce test failures and make the results unreliable.

Instead of using multiple repositories for the different versions of Firefox we handle everything inside the same repository by using multiple heads. At the moment the following heads exist in the repository:

Nightly channel: default
Aurora channel: mozilla-aurora
Beta channel: mozilla-beta
Release channel: mozilla-release
Firefox 17.0 ESR: mozilla-esr17

By cloning the repository, the default branch is selected automatically. As long as the tests will be run against a Nightly build of Firefox, that's fine. But if you want to run the tests against an older version, the head has to be switched. To check which branches exist run the command below and you will get a list with the revision ID ordered by the latest check-in.

$ hg branches
default                     2348:1397c0ccc72a
mozilla-beta                2346:710f4d67b0bb
mozilla-release             2345:6225fdae24f2
mozilla-aurora              2343:c08d3833d1c0
mozilla-esr17               2342:11b104dedf99

If you do not know which branch is actually selected, run:

$ hg branch
default

Given the output the default branch is currently selected and the tests will work with versions of Firefox Nightly builds. If another branch is needed because tests have to be run against Firefox builds on the Aurora channel, the following command switches to the aurora branch:

$ hg up -C mozilla-aurora
84 files updated, 0 files merged, 1 files removed, 0 files unresolved

The repository and all its test will be updated to the latest version of tests in that branch.

Note: According to the rapid release cycle of Firefox, code merges between the branches will happen every 6 weeks. Our branches have to follow the merge process at the same day. There are more details and step by step instructions available.

Running Mozmill tests

To get familiar with Mozmill test scripts, you can take a look at the exisiting Firefox tests from the mozmill-test repository.

To run all of our Mozmill tests you should also clone the mozmill-automation repository. It contains a bunch of scripts to trigger each individual testrun. Here an example how to start the functional tests for the given version of Firefox.

$ hg clone http://hg.mozilla.org/qa/mozmill-automation/
$ cd mozmill-automation
$ ./testrun_functional.py %path_to_firefox% --report=http://mozmill-crowd.blargon7.com/db

The testrun script will automatically clone the remote mozmill-tests repository, selects the correct named branch for the version of Firefox to test, runs all the tests, and reports results to our Mozmill dashboard.

Note: If you want to use a local version of the tests you can use the --repository option with the path added, which is supported by any of the scripts.

When you work on tests this might still not satisfy your needs because most of the time you want to only run the test you are working on right now. In such a case you should use the Mozmill command line client with one of the options as given below. A fresh profile will automatically be created so the test always runs in a clean environment. Keep in mind, however, that if you want to run multiple tests inside a folder, all those tests will be executed in the same profile. Beneath those normal tests you will also be able to run restart tests like what is needed for extension installations.

You can run the mozmill or mozmill-restart client with the --help option to get a list of available options:

$ mozmill --help
Usage: mozmill [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -b BINARY, --binary=BINARY
                        Binary path.
  -s, --shell           Start a Python shell
  --show-all            Show all test output.
  --info                Print module information
  -l LOGFILE, --logfile=LOGFILE
                        Log all events to file.
  -t TEST, --test=TEST  Run test file or directory.
  --timeout=TIMEOUT     seconds before harness timeout if no communication is
                        taking place
  -D, --debug           Debug mode
  -a ADDONS, --addons=ADDONS
                        Addons paths to install.
  --report=REPORT       Report the results. Requires url to results server.
                        Use 'stdout' for stdout.
  -u, --usecode         Use code module instead of iPython
  --show-errors         Print logger errors to the console.
  -p PROFILE, --profile=PROFILE
                        Profile path.
  -P PORT, --port=PORT  TCP port to run jsbridge on.

Three of these options are the ones you will use most:

  • The most important option is -t which specifies a single test file or a test folder and its sub folders where the tests resist.
  • The -b option is useful because it lets you run the tests against a specified version of Firefox instead of letting Mozmill find the system's default Firefox browser.
  • The --show-errors option lets you get more comprehensive error output in the shell window. Below you can find some examples specific to our mozmill-test repository for Firefox.

Run normal Mozmill tests

To run our normal Mozmill tests, use the mozmill command. Here are below some examples.

To start the default Firefox application, execute the given test, and close Firefox afterward:

$ mozmill -t tests/functional/testPreferences/testRestoreHomepageToDefault.js

To start the default Firefox application, execute all the tests in the given folder and its subfolders, and close Firefox afterward:

$ mozmill -t tests/functional/testPreferences/

To start the specified version of Firefox (Windows, Linux, or OS X), execute the given test, and close the browser afterward:

$ mozmill -t tests/functional/testPreferences/testRestoreHomepageToDefault.js -b "c:\firefox 3.5\firefox.exe"   (Windows)
$ mozmill -t tests/functional/testPreferences/testRestoreHomepageToDefault.js -b "/usr/bin/firefox"             (Linux)
$ mozmill -t tests/functional/testPreferences/testRestoreHomepageToDefault.js -b "/Applications/Firefox.app"    (Mac OS X)
Note: When using the -b option the full path to the executable has to be specified on Windows and Linux while on OS X the application bundle can be used.

Run Mozmill restart tests

Restart tests can be executed by using the mozmill-restart command. It allows you to run tests like installing an extension which need a restart to finish. For restart tests you will always specify a test folder for the -t option. It will run all the test files in that folder in an alphabetical order.

To start the system's default Firefox application, run all the tests under the given folder by restarting Firefox in between each test, and finally close Firefox, you can use the following command, for example. The same profile is used for all test files inside this folder.

$ mozmill-restart -t tests/functional/restartTests/testExtensionInstallUninstall/

To start the system's default Firefox application, run the restart tests for all sub folders, and finally close Firefox, a command like the following can be used. The same profile is only used for one subfolder; it's not shared between the different subfolders.

$ mozmill-restart -t tests/functional/restartTests/

To start the specified version of Firefox, run all the tests in the given folder by restarting Firefox in between each test, and close the browser afterward:

$ mozmill-restart -t tests/functional/restartTests/testExtensionInstallUninstall/ -b "c:\firefox 3.5\firefox.exe"   (Windows)
$ mozmill-restart -t tests/functional/restartTests/testExtensionInstallUninstall/ -b "/usr/bin/firefox"             (Linux)
$ mozmill-restart -t tests/functional/restartTests/testExtensionInstallUninstall/ -b "/Applications/Firefox.app"    (Mac OS X)
Note: When using the -b option the full path to the executable has to be specified on Windows and Linux while on OS X the application bundle is used.

Writing Mozmill tests

Now that you know how to run Mozmill tests, you can help by writing new tests or by fixing existing ones. It's not hard to do, but you have to follow some simple rules so we can guarantee long-living and understandable tests for everyone.

How to start

To make it easier for you to create your first Mozmill tests, we have prepared a couple of template files. They will help you get familiar with the license block, needed test functions, shared modules, and the proper syntax to use when writing tests. You can find these files in your local version of the test repository or online.

Some specific things to pay attention to when creating tests:

  • Please update the name and the email address in the license block.
  • Use a meaningful name for your test function; one which indicates the overall target of the test.

Logging test results

Results are logged in our tests through either of two verification objects: assert and expect.

These should both be imported into your test module (note that the exact path may differ, depending on which subdirectory your test is in):

var {assert, expect} = require("../../../lib/assertions");

Each object has the same methods, detailed below. If an assert or expect method passes, each of them will log a PASS for that verification and continue. The difference is in what happens when a test fails:

A failure in an expect method will not stop the test, but will log a FAIL to the results system. Any failed result will still cause the test to also be marked as failed overall. Examples of verifications that would usually use expect include color, non-essential item text, and other aspects of state that don't really affect anything else.

expect should be used when failure for that test result will not invalidate the rest of the test.

A failure in an assert method will not only log a FAIL, but stop the test. Examples of verifications that would usually use assert include tab or dialog presence, whether a page has loaded, and other aspects of state that completely block the test if they're not as expected.

assert should be used when failure for that test result will invalidate the rest of the test.

When possible, expect should be used so that the test will continue, both to get partial results and to provide additional context to the failure. Only use assert when continuing on failure doesn't make any sense.

assert / expect methods

ok(aValue, aMessage)

Logs a PASS if aValue is true, and a FAIL if aValue is false. Use this when you have a single true/false value to test. For comparisons between an actual and expected value, see equal()and notEqual() below.

Note that true/false is in terms of JavaScript truth when non-boolean values are used. For example, 0 and null are false, and 1 and "foo" are true.

expect.ok(button.getNode().hidden, "Button is hidden"); 

equal(aValue, aExpected, aMessage)

Logs a PASS if aValue exactly equals aExpected, FAIL otherwise. Use this for comparisons between an actual and expected value.

assert.equal(numTabs, 3, "The correct number of tabs are shown"); 

notEqual(aValue, aNotExpected, aMessage)

Logs a PASS if aValue exactly equals anything other than aNotExpected, FAIL otherwise. The most common cases for this are checking that something is not 0 or a blank string, or when checking that a text value is changing but the new value isn't predictable. For predictable values, favor an equal() comparison with the new value.

assert.notEqual(newText, oldText, "The text has changed"); 

match(aString, aRegEx, aMessage)

Logs a PASS if aString matches the regular expression given in aRegEx, FAIL otherwise.

expect.match(captionText, "/mozilla/i", "The word 'Mozilla' appears somewhere in the caption");

notMatch(aString, aRegEx, aMessage)

Logs a PASS if aString does not match the regular expression given in aRegEx, FAIL otherwise.

expect.notMatch(captionText, "/mozilla/i", "The word 'Mozilla' does not appear in the caption"); 

pass(aMessage)

Logs an unconditional PASS. This should be used extremely rarely, and only in cases where a fully custom verification structure is needed and none of the other methods make sense to use. It's almost always better to save the result as a boolean and use ok() instead.

expect.pass("If the code got here, this test is passing (for now)");

fail(aMessage)

Logs an unconditional FAIL. This should be used extremely rarely, and only in cases where a fully custom verification structure is needed and none of the other methods make sense to use. It's almost always better to save the result as a boolean and use ok() instead.

expect.fail("If the code got here, this test is failing");

Coding style

There are some coding style rules you should follow when writing new tests or contributing to existing tests. These rules help make the review process as efficient as possible and makes it easier for others to read your code.

If that was not enough information, you should take a look at the existing tests or shared modules in the Mozmill test repository.

Tips and tricks

Sometimes you will run into trouble while creating Mozmill tests. Here are some suggestions that may help you sort out the problems you might run into.

  • Get familiar with the functionality provided by Mozmill and all of our existent Shared Modules; this will ease the test creation process.
  • Use the Inspector or Recorder to create the skeleton of your test. You have to add additional steps like calls to sleep functions or element checks before the test can be run.
  • If you are using controller.open() to load a web page, a controller.waitForPageLoad() has to be used right afterward in order to prevent continuing with the test before the page finishes loading; calling controller.sleep() is not sufficient.
  • Use the controller's menu API to reach commands which are only available via the main menu. A list of existing IDs for menu items can be found in the browser-menubar.inc file. Due to our localization efforts please always use the IDs of menu items instead of their names.
  • If your test needs exactly one tab open use TabbedBrowsingAPI.closeAllTabs(controller); inside the setupModule() function.
  • If you modify preferences or other global data, make sure to reset those values inside the teardownModule() function. That will clean up the environment for the next Mozmill test.
  • Avoid using any hardcoded strings for the elementslib Lookup() function. Doing so will break Mozmill tests for localized builds. After using the inspector you have to manually remove those attributes (e.g. label or accesskey) from the element string (see the next bullet).
  • If an element can only be referenced by the elementslib Lookup() function please try to remove as many attributes as possible from each hierarchy. That will make the test more readable and can avoid failed lookups when the code in Firefox changes.

The review process

Before your test can be checked into the mozmill-test repository, you have to pass the review process. The reviewer has to learn about the test and check if everything is done correctly. In order to make the review as easy as possible, be sure your test script abides by the guidelines given above. In addition to checking the syntax and code style of the test, make sure the test runs with the command line client before requesting a review. If questions arise feel free to ask in #automation or the automation developer mailing list at any time.

Simplified patch creation

The easiest way to create a patch is by using the hg diff command bounded by two other commands. With hg add you advise Mercurial to start tracking your test file. It's needed to see your test content in the diff output. Once the patch has been created you can use hg rm to safely remove the test from the tracking list. That will guarantee that no conflicts will happen when you pull a new version to your local copy of the repository.

Imagine you have created a test called testZoomSettings.js which is saved under tests/functional/testLayout/ and you want to create a patch called patch_file:

$ hg add tests/functional/testLayout/testZoomSettings.js
$ hg diff >patch_file
$ hg rm -f tests/functional/testLayout/testZoomSettings.js

After running those commands, you will find the file patch_file in the current folder which can be uploaded as attachment to the bug report.

Advanced patch creation

As you can imagine, it's hard to track all your files when you are working on several tests in parallel, because all those files will lingering around in your working copy. To prevent that and to gain the overview you can use the Mercurial Queue extension.

In the example below, you can see how it works, starting with a new test named testZoomSettings.js:

$ hg qnew zoomsettings                                    (Add a new named patch to the queue of patches)
$ vi tests/functional/testLayout/testZoomSettings.js       (Create your test and apply the template structure)
$ hg add tests/functional/testLayout/testZoomSettings.js  (Start tracking the test file)
$ hg diff                                                 (Create a diff output of the current state)
$ hg qrefresh -m "Commit message (see below)"             (Update the patch by accepting all changes and giving a necessary commit message)
$ vi tests/functional/testLayout/testZoomSettings.js       (Continue to update your test)
$ hg diff                                                 (Create a diff against the last version of your patch)
$ hg qdiff                                                (Create a complete diff against the current version of the repository)
$ hg qrefresh                                             (Refresh the patch with the latest changes)
$ hg qpop                                                 (Pop the patch from the stack)
$ hg qpush                                                (Push the patch back to the stack)
$ hg export tip >patch_file                               (Create a patch based on the current state)

Commit message

Your commit message should follow a standard format of:

"Bug %number% - %Description%. r=%reviewer1%, r=%reviewer2%..."

Description should include a concise description of the changes made. Please do not include the branch name in the description.

Ex:

$ hg qrefresh -m "Bug 553616 - Fixing testPasteLocationBar.js to use utils clipboard clearing. r=gmealer"

Reviewing

When using the advanced way of creating a patch, all existing patches are located under .hg/patches. Before you ask for a review you should check the patch to ensure that it's valid. You can use the online review tool. The only warnings you should get should be those indicating that lines are too long. Further you should also check that the test is working as expected. The best solution here is to run the test via the appropriate testrun script and report the results to our dashboard. Mention the link to the report in the review request.

Reviews are managed in Bugzilla. So once your new test has been created, file a new bug report (see bug 479720 as example). Also add the MozTrap testcase IDs for all branches in the first comment of the bug. Finally your patch for the test has to be attached to the bug. Now you can request review from Henrik Skupin, Dave Hunt or Andreea Matei. If the bug you're working on has a mentor assigned at the "whiteboard" tag, then you can use request review from that person.

Note: Initial patches should be created for the default branch of the mozmill-test repository. Tests for older versions of Firefox will be backported after the test has been landed on the default branch.

Landing of patches

Once a patch has been reviewed and is ready for check-in, the reviewer will land the patch immediately or will add the keyword "checkin-needed" if another person has to land the patch. If you are the one who has check-in permissions and you have to land the patch, the following steps should be obeyed:

Preparation: Before you can push any patch to the repository the .hg/hgrc file of the local copy has to be updated so it contains the default-push path, which is usually an ssh connection:

[paths]
default = https://hg.mozilla.org/qa/mozmill-tests
default-push = ssh://hg.mozilla.org/qa/mozmill-tests/

Check-in:

  1. Make sure that the correct branch of the mozmill-tests repository has been selected, if not update accordingly.
  2. Run a "hg pull -u" to make sure that no other patches has been pushed since your last pull request.
  3. Download the patch to your local disk and import it via "hg qimport %patch%".
  4. Use "hg qpush" to push the patch to your queue. It will end-up on-top of your local queue. You can check with "hg tip".
  5. Run "hg out" to check if the user, the email address, and the summary has been set correctly.
  6. If the user name is not valid, update the changeset with "hg qrefresh -u "%username% <%email%>".
  7. If the summary is not valid, update the changeset with "hg qrefresh -m 'Bug %number% - %Description%. r=%reviewer1%, r=%reviewer2%...'".
  8. Run "hg qfinish tip", which removes the patch from your queue and commit the changes.
  9. Finally push the patch to the public repository with "hg push".
注記: If you want to become a committer, please review our commit policy.

Transplanting a Patch

In some instances, it will be necessary to check in a patch on several branches. Using the transplant extension makes it easy.

1. Update to the target branch

hg pull -u && hg update -C %target_branch%

2. Transplant the source changeset

hg transplant %changeset_ID%

3. Finally, push the change

hg push
NOTE: To use transplant, you need to have the transplant extension added to your .hgrc file

Backing out patches

If a new test immediately fails after its check-in, we will have to back out the responsible changeset. Follow those instructions in how to correctly do a back-out. As back-out comment use "Backed out changeset %id% due to %failure%".

Merging heads

If multiple heads have been created accidentally on a branch, those have to be merged into the original head of the given branch.

$ hg heads                                              # Check if multiple heads per branch exist
$ hg up -C %target_branch%                              # Switch to the target branch
$ hg merge -r %changeset%                               # Merge duplicate changeset
$ hg diff                                               # Check diff of merge and ask for feedback/review if necessary
$ hg commit -m "Merge %changeset% into %target_branch%  # Commit the merge and specify a comment
$ hg push                                               # Push the merge

Other types of Mozmill tests

Mozmill is also able to automate testing in various other areas. For now we cover areas like:

Document Tags and Contributors

Contributors to this page: Whimboo, ethertank
最終更新者: ethertank,