Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.
The StandaloneTalos page explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.
- You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
- Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).
- On addon.config, add any preferences and their corresponding values to the preferences variable if they are necessary to reliably test your add-on's startup performance.
- On addon.config, add a path to your XPI file to the extensions variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.
- On both files, you can remove or comment out all tests in the tests section, except for ts and ts_paint.
Running the tests
To tun the tests, close all instances of Firefox you have open.
To run the baseline tests:
python run_tests.py -s baseline.config
To run the add-on tests:
python run_tests.py -s addon.config
Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should begin similar to this:
RETURN:s: firefox_testing firefox_testing: Started Tue, 03 May 2011 12:26:31 Running test ts: Started Tue, 03 May 2011 12:26:31 Screen width/height:1920/1200 colorDepth:24 Browser inner width/height: 1024/653 i,val 0,740 1,612 2,615 3,583 4,609 5,594 6,594 7,619 8,581 9,607 10,602 11,589 12,592 13,612 14,587 15,588 16,599 17,578 18,760 19,586 RETURN: ts: 604.58
Interpreting the results
The first set of 20 measurements are the ones that correspond to startup time. This is measured from the time the application process is started until a web page is loaded.
An average of the test results (discarding the highest one) is included at the end:
RETURN: ts: 604.58
The overhead percentage as shown on AMO pages is calculated with the formula:
overhead_percentage = (addon_average - baseline_average) / baseline_average
It represents the average add-on startup overhead as a percentage of the baseline average startup time.
These are warm startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and other files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a cold startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it would be a more realistic measurement of add-on performance impact. However, testing cold startup is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.
It's also important to keep in mind that the test results are very dependent on what you're running on your system simultaneously. For more consistent results you should run these tests will no other applications open.
Using About Startup
The About Startup add-on provides a much simpler way to measure startup performance. Once installed, it will gather startup data and display it in the about:startup page. The add-on is very lightweight, so it shouldn't have a noticeable negative effect on your baseline and add-on tests.
To test using About Startup, you should set up a new Firefox profile, install this add-on, and then open and quit Firefox repeatedly. If you want to match Talos, you should do this 20 times. Then look at the results in the about:startup page. This should provide your baseline numbers. Then repeat the process with both About Startup and your add-on installed.
The about:startup page shows 3 different time measurements: main, sessionRestored and firstPaint. main happens very early in the startup process and should not be affected by add-ons being installed. The firstPaint time should be roughly similar to what Talos tests, although it has been discovered that on some systems the first page can load before this. sessionRestored is fired later in the process, when Firefox has loaded all tabs from the saved session.