Mozilla kicked off an Add-on Performance initiative that intends to make users more aware of the performance impact of the add-ons they install. This article explains how to measure add-on startup time and produce similar results as the ones on AMO.
The Performance Best Practices documentation explains how to make add-ons that have a small footprint.
Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.
The Standalone Talos page details how to download and get Talos up and running. It's not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.
- You'll need 2 config files: one for your baseline numbers (no add-ons installed), and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
- Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (e.g. /Applications/Firefox.app/Contents/MacOS/firefox-bin).
- On addon.config, add any preferences and their corresponding values to the preferences section if they are necessary to reliably test your add-on's startup performance.
- On addon.config, set the extensions variable to [ '/path/to/xpi' ].
- On both files, you should remove or comment out all tests in the tests section, except for ts.
Running the tests
To tun the tests, close all Firefox instances you have open.
To run the baseline tests:
python run_tests.py -s baseline.config
To run the add-on tests:
python run_tests.py -s addon.config
Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete.
The output should begin similar to this:
RETURN:s: firefox_testing firefox_testing: Started Tue, 03 May 2011 12:26:31 Running test ts: Started Tue, 03 May 2011 12:26:31 Screen width/height:1920/1200 colorDepth:24 Browser inner width/height: 1024/653 i,val 0,740 1,612 2,615 3,583 4,609 5,594 6,594 7,619 8,581 9,607 10,602 11,589 12,592 13,612 14,587 15,588 16,599 17,578 18,760 19,586 RETURN: ts: 604.58
Interpreting the results
The first set of 20 measurements corresponds to startup time. This is measured from the time the application process is started until a web page is loaded.
An average of the test results (discarding the highest one) is included at the end:
RETURN: ts: 604.58
The overhead percentage as shown on AMO pages is calculated with the formula:
overhead_percentage = (addon_average - baseline_average) / baseline_average
It represents the average add-on startup overhead as a percentage of the baseline average startup time.
These are warm startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a cold startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it is a more realistic measurement of add-on startup performance overhead. However, cold startup testing is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.
It's also important to keep in mind that the test results are very dependent on your system and what you're running on it. For more consistent results you should run these tests wiith no other applications open.
Using About Startup
The About Startup add-on provides a much simpler way to measure startup performance. Once installed, it will gather startup data and display it in the about:startup page. The add-on is very lightweight, so it shouldn't have a noticeable negative effect on your baseline and add-on tests.
To test with About Startup, you should set up a new Firefox profile, install this add-on, and then open and quit Firefox repeatedly. If you want to match Talos, you should do this 20 times. Then look at the results in the about:startup page. This should provide your baseline numbers. Then repeat the process with both About Startup and your add-on installed. The overhead can be calculated using the same formula.
The about:startup page shows 3 different time measurements: main, sessionRestored and firstPaint. main happens very early in the startup process and should not be affected by add-ons being installed. The firstPaint time should be roughly similar to what Talos tests. This can vary between systems, though. sessionRestored is fired later in the process, when Firefox has loaded all tabs from the saved session.