mozilla
Your Search Results

    Peptest

    Overview

    Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple JavaScript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion.

    The code is currently located at github. If you would like to help you can send pull requests, or ping ahal or mcote on IRC for ways to get started. Peptest is also mirrored to Mozilla Central. Firefox tests can be added to the tests directory (don't forget to also add them to the Test manifest).

    For a longer description of what "unresponsiveness" actually means, see ted's blog post.

    The Peptest wiki page contains project-related information, including current status and plans for future work.

    Using Peptest

    Running tests

    Standalone

    Currently tests are run from the command line with python. Make sure you have pip and virtualenv installed.

    1. Create and activate a virtualenv, then run the command
      pip install peptest
    2. Run the command (use --help for a full list of options)
      peptest --binary <path_to_binary> --test-path <path_to_test_manifest>

    Alternatively if you will be modifying peptest, you can install peptest from the source repositories.

    1. Clone the mozbase repository.
    2. Create a new virtualenv and run 'python setup_development.py' to install all the packages.
    3. Clone the Peptest repo and install peptest (python setup.py develop)
    4. Run the command (use --help for a full list of options)
      peptest --binary <path_to_binary> --test-path <path_to_test_manifest>

    From within mozilla-central

    After building, go to your obj dir and type make peptest. This will run all tests in proxy mode, i.e., without accessing the network.

    Test manifest

    All parameters are optional except for the --test-path parameter. This parameter should contain the path to a ManifestDestiny manifest that lists all the tests that will be run. Manifests should be of the following format:

    # test paths are relative to the manifest
    [test1.js]
    [test2.js]
    [foo/bar/test3.js]
    # this test is disabled and won't be run
    [test4.js]
    disabled = "Doesn't work, see bug 123456"
    

    Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.

    Fail threshold

    One more thing that can be specified from the manifest is the failure threshold. When each test finishes, the Peptest harness will aggregate all of the unresponsive times into one metric that can be used for regression detection. If you specify a failThreshold for a test, it will only be reported as a failure if its metric is higher than the threshold.

    # no failThreshold, this test will fail if there is any unresponsiveness whatsoever
    [test1.js]
    # these tests will only fail if their metrics are greater than their failThresholds
    [test2.js]
    failThreshold = 42
    [test3.js]
    failThreshold = 500
    

    Test format

    Tests are simply JavaScript files that will be executed in chrome space. This means they have access to any of the APIs that an extension would normally have access to. In addition to this, they also have access to Mozmill's driver for convenient automation of Firefox/Thunderbird's UI.

    It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called performAction(name, func) which takes in two parameters: an action name and a function pointer whose contents consist of a single action.

    The following example test will make sure that the browser remains responsive while opening a page and while searching in Google.

    // import Mozmill and initialize a controller
    Components.utils.import('resource://mozmill/driver/mozmill.js');
    let controller = getBrowserController();
    
    // Create our first action which will open Google
    pep.performAction('open_google', function() {
      controller.open('http://google.com');
      controller.waitForPageLoad();
    });
    
    // stuff not inside a performAction() call won't be tested for responsiveness
    let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
    let button = findElement.Name(controller.tabs.activeTab, 'btnK');
    
    // Create our second action which will perform a search in the google searchbox
    pep.performAction('search_google', function() {
      textbox.sendKeys('foobar');
      button.click();
      controller.waitForPageLoad();
    });
    

    For documentation on using Mozmill's driver, see the Mozmill Reference Desk.

    Note that using Mozmill may be convenient but is not required. Here is another example that tests resizing the window without Mozmill.

    // getWindow is a convenience function exposed to tests
    // It uses nsIWindowMediator to get the current window
    let window = pep.getWindow();
    let width = window.outerWidth;
    let height = window.outerHeight;
    
    pep.performAction('resize_by', function() {
      window.resizeBy(100, 100);
    });
    
    pep.performAction('resize_to', function() {
      window.resizeTo(800, 600);
    });
    
    // Tests should clean up after themselves
    window.resizeTo(width, height);
    

    See the existing Firefox peptests for more examples.

    Log format

    Peptest uses the mozlog module to standardize on the logging format.

    PEP TEST-START | test1.js
    PEP TEST-PASS  | test1.js | fail threshold: 0.0 < metric: 0.0
    PEP TEST-END   | test1.js | finished in: 1536 ms
    PEP TEST-START | test2.js
    PEP WARNING    | test2.js | loadPage | unresponsive time: 116 ms
    PEP WARNING    | test2.js | loadPage | unresponsive time: 61 ms
    PEP WARNING    | test2.js | loadPage | unresponsive time: 275 ms
    PEP TEST-UNEXPECTED-FAIL | test2.js | fail threshold: 90.0 < metric: 92.802
    PEP TEST-END   | test2.js | finished in: 8022 ms
    PEP TEST-START | test3.js
    PEP WARNING    | test3.js | scroll_menu | unresponsive time: 121 ms
    PEP TEST-PASS  | test3.js | fail threshold: 15.0 < metric: 14.641
    PEP TEST-END   | test3.js | finished in: 2137 ms
    

    When a responsiveness measurement runs over 50ms, a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.

    When the test is finished, the total run time and a metric are output. The metric is explained in the next section.

    Metric

    The metric is a number that is calculated based on all accumulated unresponsive times. The function used to calculate the metric is

    # values is a list of unresponsive values that occurred during the test
    def get_metric(values):
        return sum([x*x / 1000.0 for x in values])
    

    The metric is useful for determining whether a test has regressed or not. For example if a test used to have a metric of 56.43 then all of a sudden has a metric of 193.42 we know a regression has happened.

    Important: You cannot compare the metric from one test to that of another. Tests can be of an arbitrary length and therefore it is impossible to conclude that a test with a metric of 10 is more responsive than a test with a metric of 1000. You must only compare metric values to previous metric values of the same test.

    Proxying to local files to avoid network access

    Peptest can be configured to proxy all network requests to local files. There are three related options:

    • --proxy: This option must be given with the location to a text file specifying what URLs should be proxied. The file should have one or more lines in the form <url> [<options>]. See testing/peptest/tests/firefox/server-locations.txt for an example (as used in buildbot).
    • --server-path: This must be set to the path to a directory serving as the root of the local web server.
    • --proxy-host-dirs: This indicates that the files in the web server root should be organized according to URL. Without this option, the same content, as found in the server path, is served for all proxied URLs. With this option, the web server root should have one directory per URL. See the server files in the repo for an example of how the server root should be structured when --proxy-host-dirs is given (as used in buildbot).

    Peptest Statistics

    Peptest is run automatically as part of buildbot on all pushes to try, mozilla-inbound, and mozilla-centric. The results are gathered and displayed for analysis. (This is a temporary URL; eventually they will be folded into the next version of the graph server/perfomatic)

    API reference

    There is a Peptest API that gets injected into the scope of every test under the "pep" namespace. Each test can use the following methods:

    General methods

    pep.performAction(name, function)

    Performs an action during which responsiveness measurements are recorded.

    name
    The name of the action to run.
    function
    A function to handle running the action.

    pep.getWindow(type)

    Returns a reference to the current window.

    type
    A string specifying the XUL windowtype of the window type to return; the default is "navigator:browser".

    pep.sleep(milliseconds)

    Unfortunately, UI automation is very dependent on timing and sometimes it is necessary to sleep; however, you should avoid this method if you can.

    milliseconds
    The number of milliseconds for which to sleep.

    Logging methods

    These log messages to the console.

    pep.log.debug(msg)

    msg
    The message to output.

    pep.log.info(msg)

    msg
    The message to output.

    pep.log.warning(msg)

    msg
    The message to output.

    pep.log.error(msg)

    msg
    The message to output.

    Document Tags and Contributors

    Contributors to this page: Sheppy, mcote, mrcote
    Last updated by: Sheppy,