mozilla
Your Search Results

    Gaia performance tests Redirect 1

    This article provides information about running performance tests on Gaia, as well as how to go about creating new tests.

    Running the tests

    The tests are run on a regular basis on Datazilla; however, you can also run them yourself. To do so, you'll need an engineer build with Marionette enabled and remote debugging disabled. See Gaia Build System Primer, Customizing the preferences for more information on how doing this.

    Test Requirements

    Since bug 915156 landed on December 6th 2013, make test-perf requires Node.js on the host to run the tests. The relevant modules should be installed automatically with npm.

    Prior to running the test, you need to configure a runner host. The runner host is a module that will either run the test in B2G desktop or on a device (real or virtual — like an emulator). By default it runs in B2G desktop, which is not very relevant for performance. To configure the runner just edit the file local.mk in the Gaia top level directory (create it if it doesn't already exist) and put the following line:

    MARIONETTE_RUNNER_HOST=marionette-device-host

    This will use the device host runner. The default value is marionette-b2gdesktop-host.

    The alternative to this is to do:

    MARIONETTE_RUNNER_HOST=marionette-device-host make test-perf 

    Note: only one device at a time is supported, either an emulator or real device. Make sure you have an up-to-date Gaia version running on it.

    Output

    Note: This is available in Gaia 1.4, but you should be able to run it against Gaia 1.3 on a real device.

    The test output the data in JSON format. By default it is output to stdout and might be mixed with error message from other commands like npm. This is not a very good idea for automation. So you can redirect this JSON output to a file. Just define MOZPERFOUT for the host runner, either on the command line as an option or in the local.mk file as shown above.

    MOZPERFOUT=myfile.json

    Running all tests

    You must make sure that you run the performance test using the same version of Gaia that has been installed on the target device. There is no guarantee that it will work otherwise as some subtle changes my have been make.

    make test-perf

    Running tests for a specific app

    APP=browser make test-perf

    Running tests for a set of apps

    APPS="browser communications/contacts" make test-perf

    Setting the number of runs

    By default, each test is run five times. You can change that by setting the value of RUNS before running the tests. For example, to run each test three times you'd use this option:

    RUNS=3 make test-perf

    Writing new tests

    With the details of running the test suite out the way, let's now look at how you can write your own performance tests for Gaia.

    Startup tests

    If you only want to record events that happen during the startup of an app, there's a generic test written to help you do so. To record the events, all you have to do is dispatch them.

    First, include our helper in your app:

    <script defer src='/shared/js/performance_testing_helper.js'></script>
    

    Note: If you use a module loader like RequireJS or Alameda, you might prefer to use that, which is perfectly fine.

    You need to be cautious and make sure you adjust the unit tests so that the PerformaceTestingHelper is either loaded or shimmed. A simple shim is to put this in the unit test source file:

    var PerformanceTestingHelper = {
      dispatch: function() { }
    };
    

    The Travis CI jobs we run out of Github will error if you don't do that properly.

    Having done that, you can use the helper to dispatch events when it seems appropriate to do so:

    PerformanceTestingHelper.dispatch('foobar');
    

    When you're ready to stop collecting data and to report the numbers, you need to send the startup-path-done event to tell the helper to do so:

    PerformanceTestingHelper.dispatch('startup-path-done');

    Once you've done that, you need to whitelist your app in /tests/performance/startup_events_test.js, by adding it to the list specified by const whitelistedApps.

    Running tests from a non-engineering device

    If you don't have an engineering build on your phone you'll have to do some additional steps:

    1. Clone B2G, and build with ./config.sh DEVICE-NAME (e.g. ./config.sh keon)
    2. Build the Gecko part via ./build.sh gecko
    3. Connect the phone and flash gecko via ./flash.sh gecko
    4. Clone Gaia, and create a file build/custom-prefs.js with content user_pref("marionette.defaultPrefs.enabled", true);
    5. Enable Remote Debugging on the phone and run make reset-gaia to reset the phone (or make install-gaia if you trust yourself)
    6. Disable Remote Debugging and verify that everything is OK by running adb devices. The device should show up.
    7. Now running a perf test should work. Verify via RUNS=1 APP=browser make test-perf

    Filing bugs

    Please file bugs in Bugzilla, product "Firefox OS", component "Gaia::PerformanceTest".

    See also

    Document Tags and Contributors

    Contributors to this page: teoli
    Last updated by: teoli,