Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.
The StandaloneTalos page explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.
- You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
- Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).
- On addon.config, add any preferences and their corresponding values to the preferences variable if they are necessary to reliably test your add-on's startup performance.
- On addon.config, add a path to your XPI file to the extensions variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.
- On both files, you can remove or comment out all tests in the tests section, except for ts and ts_paint.
Running the tests
To tun the tests, close all instances of Firefox you have open.
To run the baseline tests:
python run_tests.py -s baseline.config
To run the add-on tests:
python run_tests.py -s addon.config
Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should be similar to this:
RETURN:s: firefox_testing firefox_testing: Started Tue, 03 May 2011 12:26:31 Running test ts: Started Tue, 03 May 2011 12:26:31 Screen width/height:1920/1200 colorDepth:24 Browser inner width/height: 1024/653 i,val 0,740 1,612 2,615 3,583 4,609 5,594 6,594 7,619 8,581 9,607 10,602 11,589 12,592 13,612 14,587 15,588 16,599 17,578 18,760 19,586 RETURN: ts: 604.58 i,value 0,194 1,141 2,141 3,144 4,141 5,142 6,142 7,166 8,145 9,144 10,155 11,141 12,141 13,143 14,156 15,146 16,154 17,144 18,176 19,144 RETURN: ts_shutdown: 147.68 Completed test ts: Stopped Tue, 03 May 2011 12:32:03 RETURN: cycle time: 00:05:31
firefox_testing: Stopped Tue, 03 May 2011 12:32:03
Interpreting the results
The first set of 20 measurements are the ones that correspond to startup time. An average (after discarding the highest test result) is included at the end:
RETURN: ts: 604.58
The overhead percentage as shown on AMO pages is calulated with the formula:
overhead_percentage = (addon_average - baseline_average) / baseline_average
It represents the average add-on startup overhead as a percentage of the baseline average startup time.