TPS Tests

TPS is an end to end test for Sync. It's name stands for Testing and Profiling tool for Sync (which is a misnomer, since it doesn't do any profiling), and it should not be confused with the similarly named tests in Talos.

TPS consists of a Firefox extension of the same name, along with a python test runner, both of which live inside mozilla-central. The python test runner will read a test file (in JavaScript format), setup one or more Firefox profiles with the necessary extensions and preferences, then launch Firefox and pass the test file to the extension. The extension will read the test file and perform a series of actions specified therein, such as populating a set of bookmarks, syncing to the Sync server, making bookmark modifications, etc.

A test file may contain an arbitrary number of sections, each involving the same or different profiles, so that one test file may be used to test the effect of syncing and modifying a common set of data (from a single Sync account) over a series of different events and clients.

Set up an environment and run a test

To run TPS, you should create a new firefox account using a restmail.net email address (Strictly speaking, restmail isn't required, but it will allow TPS to automatically do account confirmation steps for you. Even if you opt not to use restmail, do not use your personal firefox account, as TPS will delete and replace the data in it many times, not to mention the first run is very likely to fail, since it expects a clean start).

Note: Be prepared not to use your computer for 15 or so minutes after starting a full run of TPS, as it will open and close a fairly large number of Firefox windows.

Steps

  1. Get the source code

    Clone mozilla-central (choose your flavor):

    • hg clone hg.mozilla.org/mozilla-central
      or
    • git clone github.com/mozilla/gecko-dev
  2. cd into the tps folder

    cd testing/tps
  3. Create the environment

    I suggest the path to be outside of the mc source tree

    python create_venv.py --username=%EMAIL% --password=%PASSWORD% %PATH%
  4. Activate the environment

    source %PATH%/bin/activate
  5. Run some tests

    Note that the testfile is NOT a path, it should only be the filename from services/sync/tests/tps/

    runtps --debug --testfile %TEST_FILE_NAME% --binary %FIREFOX_BINARY_PATH%
    1. Additionally, omitting a --testfile parameter will cause it to run all TPS tests listed in services/sync/tests/tps/all_tests.json
    2. You can also prefix with MOZ_HEADLESS=1 to run in headless mode (recommended)
An example on OSX, for headlessly running just the test_sync.js testfile against a locally built firefox (where the mozconfig set the objdir to obj-ff-artifact):
 
MOZ_HEADLESS=1 runtps --debug --testfile test_sync.js --binary obj-ff-artifact/dist/Nightly.app/Contents/MacOS/firefox
 

Running TPS against stage, or dev FxA

TPS can be configured using the $TPS_VENV_PATH/config.json file. In particular, it will set preferences from the "preferences" property, and so you can set the "identity.fxaccounts.autoconfig.uri" preference to point to any FxA server you want. For example, a (partial) tps config for testing against stage might look like:
 
{
  // ...
  "fx_account": {
    "username": "foobar@restmail.net",
    "password": "hunter2"
  },
  "preferences": {
    // use "https://stable.dev.lcip.org" for dev instead of stage
    "identity.fxaccounts.autoconfig.uri": "https://accounts.stage.mozaws.net"
    // possibly more preferences...
  },
  // ...
}
Note that in this example, the foobar@restmail.net account must be registered on stage, otherwise authentication will fail (and the whole test will fail as well.  You can sign up for an FxA account on stage or dev by creating an FxA account after adding the identity.fxaccounts.autoconfig.uri preference (with the appropriate value) to about:config. Additionally, note that the config file must parse as valid JSON, and so you can't have comments in it (sorry, I know this is annoying). One alternative is to put underscores before the "disabled" preferences, e.g. "_identity.fxaccounts.autoconfig.uri": "...".

Writing TPS tests.

The easiest way to understand this is to look at a simple TPS test file.

/*
 * The list of phases mapped to their corresponding profiles.  The object
 * here must be in strict JSON format, as it will get parsed by the Python
 * testrunner (no single quotes, extra comma's, etc).
 */

var phases = { "phase1": "profile1",
               "phase2": "profile2",
               "phase3": "profile1" };

/*
 * Bookmark asset lists: these define bookmarks that are used during the test
 */

// the initial list of bookmarks to be added to the browser
var bookmarks_initial = {
  "menu": [
    { uri: "http://www.google.com",
      title "google.com",
      changes: {
        title: "Google"
      }
    },
    { folder: "foldera" },
    { folder: "folderb" }
  ],
  "menu/foldera": [
    { uri: "http://www.yahoo.com",
      title: "testing Yahoo",
      changes: {
        location: "menu/folderb"
      }
    }
  ]
};

// the state of bookmarks after the first 'modify' action has been performed
// on them
var bookmarks_after_first_modify = {
  "menu": [
    { uri: "http://www.google.com",
      title "Google"
    },
    { folder: "foldera" },
    { folder: "folderb" }
  ],
  "menu/folderb": [
    { uri: "http://www.yahoo.com",
      title: "testing Yahoo"
    }
  ]
};

/*
 * Test phases
 */

Phase('phase1', [
  [Bookmarks.add, bookmarks_initial],
  [Sync, SYNC_WIPE_SERVER]
]);

Phase('phase2', [
  [Sync],
  [Bookmarks.verify, bookmarks_initial],
  [Bookmarks.modify, bookmarks_initial],
  [Bookmarks.verify, bookmarks_after_first_modify],
  [Sync]
]);

Phase('phase3', [
  [Sync],
  [Bookmarks.verify, bookmarks_after_first_modify]
]);

 

The effects of this test file will be:

  1. Firefox is launched with profile1, the TPS extension adds the two bookmarks specified in the bookmarks_initial array, then they are synced to the Sync server. The SYNC_WIPE_SERVER argument causes TPS to set the firstSync="wipeServer" pref, in case the Sync account already contains data. Firefox closes.
  2. Firefox is launched with profile2, and all data is synced from the Sync server. The TPS extension verifies that all bookmarks in the bookmarks_initial list are present. Then it modifies those bookmarks by applying the "changes" property to each of them. E.g., the title of the first bookmark is changed from "google.com" to "Google". Next, the changes are synced to the Sync server. Finally, Firefox closes.
  3. Firefox is launched with profile1 again, and data is synced from the Sync server. The TPS extension verifies that the bookmarks in bookmarks_after_first_modify list are present; i.e., all the changes performed in profile2 have successfully been synced to profile1. Lastly, Firefox closes and the tests ends.

Test format

TPS tests are written in JavaScript, with a particular syntax.  Each test has three sections:

  1. A list of test phases.
  2. A list of static or dynamically created asset lists, which contain the bookmarks, history entries, form data values, etc, that will be used by the test.
  3. A list of phases to execute.

Test phases

The test phase list looks like this:

/*
 * The list of phases mapped to their corresponding profiles.  The object
 * here must be in strict JSON format, as it will get parsed by the Python
 * testrunner (no single quotes, extra comma's, etc).
 */

var phases = { "phase1": "profile1",
               "phase2": "profile2",
               "phase3": "profile1",
               "phase4": "profile2" };

This section maps test phases to profiles.  The profile names are arbitrary; they will be generated by the TPS testrunner.  The phase names much match the names of the test phases defined in the phase section.  As noted in the comments above, this section must be in strict JSON format, as it will get parsed by the Python testrunner.

Asset lists

A test file will contain one or more asset lists, which are lists of bookmarks, passwords, or other types of browser data that are relevant to Sync. The format of these asset lists vary depending on asset type, and are explained in detail at these links:

Test Phases

The phase blocks are where the action happens! They tell TPS what to do. Each phase block contains the name of a phase, and a list of actions. TPS iterates through the phase blocks in alphanumeric order, and for each phase, it does the following:

  1. Launches Firefox with the profile from the phases object that corresponds to this test phase.
  2. Performs the specified actions in sequence.
  3. Determines if the phase passed or failed; if it passed, it continues to the next phase block and repeats the process.

A phase is defined by calling the Phase function with the name of the phase and a list of actions to perform:

Phase('phase1', [
  [Bookmarks.add, bookmarks_initial],
  [Passwords.add, passwords_initial],
  [History.add, history_initial],
  [Sync, SYNC_WIPE_SERVER],
]); 

Each action is an array, the first member of which is a function reference to call, the other members of which are parameters to pass to the function.  Each type of asset list has a number of built-in functions you can call, described in the section on Asset lists; there are also some additional built-in functions.

Built-in functions

Sync(options)

Initiates a Sync operation.  If no options are passed, a default sync operation is performed.  Otherwise, a special sync can be performed if one of the following are passed:  SYNC_WIPE_SERVER, SYNC_WIPE_CLIENT, SYNC_RESET_CLIENT.

Logger.logInfo(msg)

Logs the given message to the TPS log.

Logger.AssertTrue(condition, msg)

Asserts that condition is true, otherwise an exception is thrown and the test fails.

Logger.AssertEqual(val1, val2, msg)

Asserts that val1 is equal to val2, otherwise an exception is thrown and the test fails.

Custom functions

You can also write your own functions to be called as actions.  For example, consider the first action in the phase above:

[Bookmarks.add, bookmarks_initial]

You could rewrite this as a custom function so as to add some custom logging:

[async () => {
  Logger.logInfo("adding bookmarks_initial");
  await Bookmarks.add(bookmarks_initial);
}]

Note that this is probably best used for debugging, and new tests that want custom behavior should add it to the TPS addon so that other tests can use it.

Troubleshooting and debugging tips for writing and running TPS tests

  1. TPS evaluates the whole file in every phase, so any syntax error(s) in the file will get reported in phase 1, even though the error may not be in phase 1 itself.
  2. Inspect tps.log. When a tps test fails, the log is dumped to tps.log in the virtualenv.
  3. Inspect about:sync-log. Every sync should have a log and every item synced should have a record.
  4. Run runtps with --debug. This will enable much more verbose logging in all engines.
  5. run test_sync.js. This test generally validates your tps setup and does a light test of a few engines.
  6. Comment out the goQuitApplication() calls in services/sync/tps/extensions/tps/modules/tps.jsm (remember to undo this later!).
    1. You will have to manually quit the browser at each phase, but you will be able to inspect the browser state manually.
    2. Using this technique in conjunction with aboutsync is helpful. (Note that the python testrunner will generally still kill firefox after a TPS test runs for 5 or so minutes, so it's often helpful to kill the python testrunner outright, and then use aboutsync in a different instance of the browser).
  7. A TPS failure may not point directly to the problem. For example,
    1. Most errors involving bookmarks look like "Places Item not found in expected index", which could mean a number of issues. The other engines are similarly unhelpful, and will likely fail if there's any problem, without really indicating what the problem is.
    2. It's common for the phase after the problem to be the one reporting errors (e.g. if one phase doesn't upload what it should, we won't notice until the next phase).
  8. TPS runs one "cleanup" phase for each profile (even for failed tests), which means most tests have two cleanup phases. This has a couple side effects
    1. You usually need to scroll up a bit in the log past the end of the test to find the actual failure.
    2. When one of the cleanup phases fails (often possible if firefox crashes or TPS hangs), there's no guarantee that the data was properly cleaned up, and so the next TPS test you run may fail due to the leftover data. This means you may want to run a TPS test twice to see if it really failed, or it just started with garbage.
  9. Feel free to ask for help with setting up and running TPS in the #sync IRC channel!

Document Tags and Contributors

 Contributors to this page: tcsc, teoli, andrei.eftimie, ally@mozilla.com, Sheppy, jgriffin, gps
 Last updated by: tcsc,