mozilla

Revision 64172 of TPS

  • Revision slug: TPS
  • Revision title: TPS
  • Revision id: 64172
  • Created:
  • Creator: jgriffin
  • Is current revision? No
  • Comment one or more formatting changes
Tags: 

Revision Content

TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.

Installation

  hg clone http://hg.mozilla.org/services/tps
  cd tps
  python setup.py develop

Because TPS currently utilizes a custom version of Mozmill, you may want to execute the above inside a virtualenv.

Running Tests

TPS can be invoked using the 'runtps' command from the tps directory.  Command arguments are:

$ runtps --help
Usage: runtps-script.py [options]

Options:
  -h, --help            show this help message and exit
  --email-results       email the test results to the recipients defined in
                        the config file
  --mobile              run with mobile settings
  --autolog             post results to Autolog
  --testfile=TESTFILE   path to the test file to run [default: test_sync.js]
  --logfile=LOGFILE     path to the log file [default: tps.log]
  --binary=BINARY       path to the Firefox binary, specified either as a
                        local file or a url; if omitted, the PATH will be
                        searched;
  --configfile=CONFIGFILE
                        path to the config file to use [default: config.json]
  --pulsefile=PULSEFILE
                        path to file containing a pulse message in json format
                        that you want to inject into the monitor

For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:

runtps --testfile=test_sync.js --binary=http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-linux/1309462661/firefox-7.0a1.en-US.linux-i686.tar.bz2

To run a different test against a local build, you might use this command:

runtps --testfile=tests/test_client_wipe.js --binary=~/builds/services-central/firefox

Tests are specified relative to the tps directory.  Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.

The config file

Several options are set using a configuration file, which is config.json by default.  This is a JSON file with the following format:

{
  "account": {
    "serverURL": "",
    "admin-secret": "",
    "username": "crossweaveservices@mozilla.com",
    "password": "*********",
    "passphrase": "r-jwcbc-dfabd-fjn72-p5vpp-iypmi"
  },
  "stageaccount": {
    "serverURL": "https://stage-auth.services.mozilla.com/",
    "username": "tpsstage@mozilla.com",
    "password": "********",
    "passphrase": "4-n5u4z-cva9e-j79mh-lj235-py3h4"
  },
  "email": {
    "username": "crossweave@mozilla.com",
    "password": "*******",
    "passednotificationlist": ["jgriffin@mozilla.com"],
    "notificationlist": ["jgriffin@mozilla.com"]
  },
  "platform": "win32",
  "os": "win7"
}
  • account - these settings specify the Sync account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use.
    • serverURL - the sync server; leave empty to use the default server
    • admin-secret - a secret phrase used to bypass captcha requirements during sync account creation; normally left empty
    • username, password, passphrase - the details of the sync account to use for testing
  • stageaccount - these settings specify a separate account on the staging Sync server that will be used for testing.  These settings are only used when test runs are automatically initiated by incoming Pulse messages.
  • email - these settings specify details of the email account used to send email notifications of test results.  These settings are only used when runtps is launched with the --email-results option.
    • username/password - account credentials of the mozilla.com email address used to send email notifications
    • passednotificationlist - list of email addresses to notify on completion of successful test runs
    • notificationlist - list of email addresses to notify on completion of failed test runs
  • platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64
  • os - free-form OS string

Manifest files

You can run multiple tests in sequence by specifying a test manifest file with the --testfile option.  Test manifests are JSON files which contain a list of test files.  There is one such file, test_manifest.json, in the TPS repo:

{ "tests": [
    "test_sync.js",
    "unittests/test_prefs.js",
    "unittests/test_tabs.js",
    "unittests/test_passwords.js",
    "unittests/test_history.js",
    "unittests/test_formdata.js",
    "tests/test_bug530717.js",
    "tests/test_bug531489.js",
    "tests/test_bug538298.js",
    "tests/test_bug556509.js",
    "tests/test_bug562515.js",
    "tests/test_bug563989.js",
    "tests/test_bug535326.js",
    "tests/test_bug501528.js",
    "tests/test_bug575423.js",
    "tests/test_bug546807.js",
    "tests/test_history_collision.js",
    "tests/test_privbrw_formdata.js",
    "tests/test_privbrw_passwords.js",
    "tests/test_privbrw_tabs.js",
    "tests/test_bookmarks_in_same_named_folder.js",
    "tests/test_client_wipe.js",
    "tests/test_special_tabs.js"
  ]
}

 

Continuous integration testing

If you run TPS without specifying a binary, TPS will start a Mozilla Pulse listener, which waits for messages from Pulse that indicate that a tinderbox build has been made.  When it receives such a message, it downloads and installs the build, then initiates three sets of tests using the test_manifest.json file.

  • Set 1:  Default Sync server, desktop prefs
  • Set 2:  Default Sync server, mobile prefs
  • Set 3:  Staging Sync server, desktop prefs

It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build.  You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:

{
  "buildtype": "opt",
  "tree": "services-central",
  "buildurl": "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}

After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.

Writing TPS Tests

TPS Tests

Revision Source

<p>TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.</p>
<h2>Installation</h2>
<pre>  hg clone http://hg.mozilla.org/services/tps
  cd tps
  python setup.py develop
</pre>
<p>Because TPS currently utilizes a custom version of Mozmill, you may want to execute the above inside a <a class=" external" href="http://www.virtualenv.org/en/latest/" title="http://www.virtualenv.org/en/latest/">virtualenv</a>.</p>
<h2>Running Tests</h2>
<p>TPS can be invoked using the 'runtps' command from the tps directory.  Command arguments are:</p>
<pre>$ runtps --help
Usage: runtps-script.py [options]

Options:
  -h, --help            show this help message and exit
  --email-results       email the test results to the recipients defined in
                        the config file
  --mobile              run with mobile settings
  --autolog             post results to Autolog
  --testfile=TESTFILE   path to the test file to run [default: test_sync.js]
  --logfile=LOGFILE     path to the log file [default: tps.log]
  --binary=BINARY       path to the Firefox binary, specified either as a
                        local file or a url; if omitted, the PATH will be
                        searched;
  --configfile=CONFIGFILE
                        path to the config file to use [default: config.json]
  --pulsefile=PULSEFILE
                        path to file containing a pulse message in json format
                        that you want to inject into the monitor
</pre>
<p>For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:</p>
<pre>runtps --testfile=test_sync.js --binary=http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-linux/1309462661/firefox-7.0a1.en-US.linux-i686.tar.bz2
</pre>
<p>To run a different test against a local build, you might use this command:</p>
<pre>runtps --testfile=tests/test_client_wipe.js --binary=~/builds/services-central/firefox
</pre>
<p>Tests are specified relative to the tps directory.  Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.</p>
<h4>The config file</h4>
<p>Several options are set using a configuration file, which is config.json by default.  This is a JSON file with the following format:</p>
<pre>{
  "account": {
    "serverURL": "",
    "admin-secret": "",
    "username": "crossweaveservices@mozilla.com",
    "password": "*********",
    "passphrase": "r-jwcbc-dfabd-fjn72-p5vpp-iypmi"
  },
  "stageaccount": {
    "serverURL": "https://stage-auth.services.mozilla.com/",
    "username": "tpsstage@mozilla.com",
    "password": "********",
    "passphrase": "4-n5u4z-cva9e-j79mh-lj235-py3h4"
  },
  "email": {
    "username": "crossweave@mozilla.com",
    "password": "*******",
    "passednotificationlist": ["jgriffin@mozilla.com"],
    "notificationlist": ["jgriffin@mozilla.com"]
  },
  "platform": "win32",
  "os": "win7"
}
</pre>
<ul> <li>account - these settings specify the Sync account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use. <ul> <li>serverURL - the sync server; leave empty to use the default server</li> <li>admin-secret - a secret phrase used to bypass captcha requirements during sync account creation; normally left empty</li> <li>username, password, passphrase - the details of the sync account to use for testing</li> </ul> </li> <li>stageaccount - these settings specify a separate account on the staging Sync server that will be used for testing.  These settings are only used when test runs are automatically initiated by incoming Pulse messages.</li> <li>email - these settings specify details of the email account used to send email notifications of test results.  These settings are only used when runtps is launched with the --email-results option. <ul> <li>username/password - account credentials of the mozilla.com email address used to send email notifications</li> <li>passednotificationlist - list of email addresses to notify on completion of successful test runs</li> <li>notificationlist - list of email addresses to notify on completion of failed test runs</li> </ul> </li> <li>platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64</li> <li>os - free-form OS string</li>
</ul>
<h4>Manifest files</h4>
<p>You can run multiple tests in sequence by specifying a test manifest file with the --testfile option.  Test manifests are JSON files which contain a list of test files.  There is one such file, test_manifest.json, in the TPS repo:</p>
<pre>{ "tests": [
    "test_sync.js",
    "unittests/test_prefs.js",
    "unittests/test_tabs.js",
    "unittests/test_passwords.js",
    "unittests/test_history.js",
    "unittests/test_formdata.js",
    "tests/test_bug530717.js",
    "tests/test_bug531489.js",
    "tests/test_bug538298.js",
    "tests/test_bug556509.js",
    "tests/test_bug562515.js",
    "tests/test_bug563989.js",
    "tests/test_bug535326.js",
    "tests/test_bug501528.js",
    "tests/test_bug575423.js",
    "tests/test_bug546807.js",
    "tests/test_history_collision.js",
    "tests/test_privbrw_formdata.js",
    "tests/test_privbrw_passwords.js",
    "tests/test_privbrw_tabs.js",
    "tests/test_bookmarks_in_same_named_folder.js",
    "tests/test_client_wipe.js",
    "tests/test_special_tabs.js"
  ]
}
</pre>
<p> </p>
<h4>Continuous integration testing</h4>
<p>If you run TPS without specifying a binary, TPS will start a <a class=" external" href="http://pulse.mozilla.org/" title="http://pulse.mozilla.org/">Mozilla Pulse</a> listener, which waits for messages from Pulse that indicate that a tinderbox build has been made.  When it receives such a message, it downloads and installs the build, then initiates three sets of tests using the test_manifest.json file.</p>
<ul> <li>Set 1:  Default Sync server, desktop prefs</li> <li>Set 2:  Default Sync server, mobile prefs</li> <li>Set 3:  Staging Sync server, desktop prefs</li>
</ul>
<p>It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build.  You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:</p>
<pre>{
  "buildtype": "opt",
  "tree": "services-central",
  "buildurl": "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}
</pre>
<p>After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.</p>
<h2>Writing TPS Tests</h2>
<p><a href="/en/TPS_Tests" title="en/TPS Tests">TPS Tests</a></p>
Revert to this revision