mozilla

Revision 64177 of TPS

  • Revision slug: TPS
  • Revision title: TPS
  • Revision id: 64177
  • Created:
  • Creator: ally@mozilla.com
  • Is current revision? No
  • Comment 14 words added
Tags: 

Revision Content

TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.

Installation

TPS comes with an installation script that sets up TPS inside a virtualenv:

  hg clone http://hg.mozilla.org/services/services-central
  cd services-central/testing/tps
  ./INSTALL.sh /path/to/create/virtualenv

Once installed, you will need to activate the virtualenv before using TPS. The command for doing this is:

  source /path/to/virtualenv/bin/activate

or, on Windows (using Mozilla build tools/msys):

  source /path/to/virtualenv/Scripts/activate

You can then launch 'runtps' to run TPS.

To quit the virtualenv, type 'deactivate' at the virtualenv prompt.

Running Tests

TPS can be invoked using the 'runtps' command.  Command arguments are:

$ runtps --help
Usage: runtps-script.py [options]

Options:
  -h, --help            show this help message and exit
  --email-results       email the test results to the recipients defined in
                        the config file
  --mobile              run with mobile settings
  --autolog             post results to Autolog
  --testfile=TESTFILE   path to the test file to run [default: test_sync.js]
  --logfile=LOGFILE     path to the log file [default: tps.log]
  --binary=BINARY       path to the Firefox binary, specified either as a
                        local file or a url; if omitted, the PATH will be
                        searched;
  --configfile=CONFIGFILE
                        path to the config file to use [default: config.json]
  --pulsefile=PULSEFILE
                        path to file containing a pulse message in json format
                        that you want to inject into the monitor

For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:

runtps --testfile=/path/to/services-central/services/sync/tests/tps/test_sync.js --binary=http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-linux/1309462661/firefox-7.0a1.en-US.linux-i686.tar.bz2

To run a different test against a local build, you might use this command:

runtps --testfile=/path/to/services-central/services/sync/tests/tps/test_client_wipe.js --binary=~/builds/services-central/firefox

Tests are specified relative to the tps directory.  Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.

The config file

Several options are set using a configuration file, which is config.json by default.  This is located inside the virtualenv created when you ran INSTALL.sh.  On Linux, this path to this file will look something like:

/path/to/virtualenv/lib/python2.6/site-packages/tps-0.2.40-py2.6.egg/tps/config.json

On Windows, it will be something like:

/path/to/virtualenv/Lib/site-packages/tps-0.2.40-py2.6.egg/tps/config.json

Be sure to edit this file, and not config.json.in in the TPS source.

config.json is a JSON file with the following format:

{
  "account": {
    "serverURL": "",
    "admin-secret": "",
    "username": "crossweaveservices@mozilla.com",
    "password": "*********",
    "passphrase": "r-jwcbc-dfabd-fjn72-p5vpp-iypmi"
  },
  "stageaccount": {
    "serverURL": "https://stage-auth.services.mozilla.com/",
    "username": "tpsstage@mozilla.com",
    "password": "********",
    "passphrase": "4-n5u4z-cva9e-j79mh-lj235-py3h4"
  },
  "email": {
    "username": "crossweave@mozilla.com",
    "password": "*******",
    "passednotificationlist": ["jgriffin@mozilla.com"],
    "notificationlist": ["jgriffin@mozilla.com"]
  },
  "platform": "win32",
  "os": "win7",
  "testdir": "/c/mozilla/services-central/src/services/sync/tests/tps/",
  "extensiondir": "/c/mozilla/services-central/src/services/sync/tps/extensions/"
 }
  • account - these settings specify the Sync account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use.
    • serverURL - the sync server; leave empty to use the default server
    • admin-secret - a secret phrase used to bypass captcha requirements during sync account creation; normally left empty
    • username, password, passphrase - the details of the sync account to use for testing
  • stageaccount - these settings specify a separate account on the staging Sync server that will be used for testing.  These settings are only used when test runs are automatically initiated by incoming Pulse messages.
  • email - these settings specify details of the email account used to send email notifications of test results.  These settings are only used when runtps is launched with the --email-results option.
    • username/password - account credentials of the mozilla.com email address used to send email notifications
    • passednotificationlist - list of email addresses to notify on completion of successful test runs
    • notificationlist - list of email addresses to notify on completion of failed test runs
  • platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64
  • os - free-form OS string
  • testdir - absolute path to the TPS tests inside the services-central tree, this value is generated by the INSTALL.sh script
  • extensiondir - absolute path to the TPS extensions inside the services-central tree, this value is generated by the INSTALL.sh script

Manifest files

You can run multiple tests in sequence by specifying a test manifest file with the --testfile option.  Test manifests are JSON files which contain a list of test files.  There is one such file, test_manifest.json, in the TPS repo:

{ "tests": [
    "test_sync.js",
    "unittests/test_prefs.js",
    "unittests/test_tabs.js",
    "unittests/test_passwords.js",
    "unittests/test_history.js",
    "unittests/test_formdata.js",
    "tests/test_bug530717.js",
    "tests/test_bug531489.js",
    "tests/test_bug538298.js",
    "tests/test_bug556509.js",
    "tests/test_bug562515.js",
    "tests/test_bug563989.js",
    "tests/test_bug535326.js",
    "tests/test_bug501528.js",
    "tests/test_bug575423.js",
    "tests/test_bug546807.js",
    "tests/test_history_collision.js",
    "tests/test_privbrw_formdata.js",
    "tests/test_privbrw_passwords.js",
    "tests/test_privbrw_tabs.js",
    "tests/test_bookmarks_in_same_named_folder.js",
    "tests/test_client_wipe.js",
    "tests/test_special_tabs.js"
  ]
}

 

Continuous integration testing

If you run TPS without specifying a binary, TPS will start a Mozilla Pulse listener, which waits for messages from Pulse that indicate that a tinderbox build has been made.  When it receives such a message, it downloads and installs the build, then initiates three sets of tests using the test_manifest.json file.

  • Set 1:  Default Sync server, desktop prefs
  • Set 2:  Default Sync server, mobile prefs
  • Set 3:  Staging Sync server, desktop prefs

It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build.  You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:

{
  "buildtype": "opt",
  "tree": "services-central",
  "buildurl": "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}

After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.

Writing TPS Tests

TPS Tests

Revision Source

<p>TPS, or Testing and Profiling tool for Sync, is a multi-profile test automation framework for Firefox Sync.</p>
<h2>Installation</h2>
<p>TPS comes with an installation script that sets up TPS inside a virtualenv:</p>
<pre>  hg clone http://hg.mozilla.org/services/services-central
  cd services-central/testing/tps
  ./INSTALL.sh /path/to/create/virtualenv
</pre>
<p>Once installed, you will need to activate the virtualenv before using TPS. The command for doing this is:</p>
<pre>  source /path/to/virtualenv/bin/activate
</pre>
<p>or, on Windows (using Mozilla build tools/msys):</p>
<pre>  source /path/to/virtualenv/Scripts/activate
</pre>
<p>You can then launch 'runtps' to run TPS.</p>
<p>To quit the virtualenv, type 'deactivate' at the virtualenv prompt.</p>
<h2>Running Tests</h2>
<p>TPS can be invoked using the 'runtps' command.  Command arguments are:</p>
<pre>$ runtps --help
Usage: runtps-script.py [options]

Options:
  -h, --help            show this help message and exit
  --email-results       email the test results to the recipients defined in
                        the config file
  --mobile              run with mobile settings
  --autolog             post results to Autolog
  --testfile=TESTFILE   path to the test file to run [default: test_sync.js]
  --logfile=LOGFILE     path to the log file [default: tps.log]
  --binary=BINARY       path to the Firefox binary, specified either as a
                        local file or a url; if omitted, the PATH will be
                        searched;
  --configfile=CONFIGFILE
                        path to the config file to use [default: config.json]
  --pulsefile=PULSEFILE
                        path to file containing a pulse message in json format
                        that you want to inject into the monitor
</pre>
<p>For example, to run the test test_sync.js using a specific tinderbox bulid, you would use this command:</p>
<pre>runtps --testfile=/path/to/services-central/services/sync/tests/tps/test_sync.js --binary=http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-linux/1309462661/firefox-7.0a1.en-US.linux-i686.tar.bz2
</pre>
<p>To run a different test against a local build, you might use this command:</p>
<pre>runtps --testfile=/path/to/services-central/services/sync/tests/tps/test_client_wipe.js --binary=~/builds/services-central/firefox
</pre>
<p>Tests are specified relative to the tps directory.  Binaries can either be local files, or http: or ftp: url's, in which case the binary is downloaded and installed prior to running the test.</p>
<h4>The config file</h4>
<p>Several options are set using a configuration file, which is config.json by default.  This is located inside the virtualenv created when you ran INSTALL.sh.  On Linux, this path to this file will look something like:</p>
<pre>/path/to/virtualenv/lib/python2.6/site-packages/tps-0.2.40-py2.6.egg/tps/config.json
</pre>
<p>On Windows, it will be something like:</p>
<pre>/path/to/virtualenv/Lib/site-packages/tps-0.2.40-py2.6.egg/tps/config.json
</pre>
<p>Be sure to edit this file, and not config.json.in in the TPS source.</p>
<p>config.json is a JSON file with the following format:</p>
<pre>{
  "account": {
    "serverURL": "",
    "admin-secret": "",
    "username": "crossweaveservices@mozilla.com",
    "password": "*********",
    "passphrase": "r-jwcbc-dfabd-fjn72-p5vpp-iypmi"
  },
  "stageaccount": {
    "serverURL": "https://stage-auth.services.mozilla.com/",
    "username": "tpsstage@mozilla.com",
    "password": "********",
    "passphrase": "4-n5u4z-cva9e-j79mh-lj235-py3h4"
  },
  "email": {
    "username": "crossweave@mozilla.com",
    "password": "*******",
    "passednotificationlist": ["jgriffin@mozilla.com"],
    "notificationlist": ["jgriffin@mozilla.com"]
  },
  "platform": "win32",
  "os": "win7",
  "testdir": "/c/mozilla/services-central/src/services/sync/tests/tps/",
  "extensiondir": "/c/mozilla/services-central/src/services/sync/tps/extensions/"
 }
</pre>
<ul> <li>account - these settings specify the Sync account that will be used during testing.  This account will be wiped during testing, so do not use an account that you use for personal use. <ul> <li>serverURL - the sync server; leave empty to use the default server</li> <li>admin-secret - a secret phrase used to bypass captcha requirements during sync account creation; normally left empty</li> <li>username, password, passphrase - the details of the sync account to use for testing</li> </ul> </li> <li>stageaccount - these settings specify a separate account on the staging Sync server that will be used for testing.  These settings are only used when test runs are automatically initiated by incoming Pulse messages.</li> <li>email - these settings specify details of the email account used to send email notifications of test results.  These settings are only used when runtps is launched with the --email-results option. <ul> <li>username/password - account credentials of the mozilla.com email address used to send email notifications</li> <li>passednotificationlist - list of email addresses to notify on completion of successful test runs</li> <li>notificationlist - list of email addresses to notify on completion of failed test runs</li> </ul> </li> <li>platform - the platform of the machine running the test, should be one of: win32, win64, linux, linux64, macosx, macosx64</li> <li>os - free-form OS string</li> <li>testdir - absolute path to the TPS tests inside the services-central tree, this value is generated by the INSTALL.sh script</li> <li>extensiondir - absolute path to the TPS extensions inside the services-central tree, this value is generated by the INSTALL.sh script</li>
</ul>
<h4>Manifest files</h4>
<p>You can run multiple tests in sequence by specifying a test manifest file with the --testfile option.  Test manifests are JSON files which contain a list of test files.  There is one such file, test_manifest.json, in the TPS repo:</p>
<pre>{ "tests": [
    "test_sync.js",
    "unittests/test_prefs.js",
    "unittests/test_tabs.js",
    "unittests/test_passwords.js",
    "unittests/test_history.js",
    "unittests/test_formdata.js",
    "tests/test_bug530717.js",
    "tests/test_bug531489.js",
    "tests/test_bug538298.js",
    "tests/test_bug556509.js",
    "tests/test_bug562515.js",
    "tests/test_bug563989.js",
    "tests/test_bug535326.js",
    "tests/test_bug501528.js",
    "tests/test_bug575423.js",
    "tests/test_bug546807.js",
    "tests/test_history_collision.js",
    "tests/test_privbrw_formdata.js",
    "tests/test_privbrw_passwords.js",
    "tests/test_privbrw_tabs.js",
    "tests/test_bookmarks_in_same_named_folder.js",
    "tests/test_client_wipe.js",
    "tests/test_special_tabs.js"
  ]
}
</pre>
<p> </p>
<h4>Continuous integration testing</h4>
<p>If you run TPS without specifying a binary, TPS will start a <a class="external" href="http://pulse.mozilla.org/" title="http://pulse.mozilla.org/">Mozilla Pulse</a> listener, which waits for messages from Pulse that indicate that a tinderbox build has been made.  When it receives such a message, it downloads and installs the build, then initiates three sets of tests using the test_manifest.json file.</p>
<ul> <li>Set 1:  Default Sync server, desktop prefs</li> <li>Set 2:  Default Sync server, mobile prefs</li> <li>Set 3:  Staging Sync server, desktop prefs</li>
</ul>
<p>It's also possible artificially inject a "fake" Pulse message into TPS, so that it will run the above set of tests against an arbitrary build.  You can specify the path to this fake Pulse message using the --pulsefile option; such files look like this:</p>
<pre>{
  "buildtype": "opt",
  "tree": "services-central",
  "buildurl": "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/services-central-win32/1309404226/firefox-7.0a1.en-US.win32.zip"
}
</pre>
<p>After completing these tests against the build specified in the pulse file, TPS will listen for real Pulse messages.</p>
<h2>Writing TPS Tests</h2>
<p><a href="/en/TPS_Tests" title="en/TPS Tests">TPS Tests</a></p>
Revert to this revision