mozilla

Revision 85705 of Measuring Add-on Startup Performance

  • Revision slug: User:Jorge.villalobos/Testing_Add-on_Startup_Performance
  • Revision title: Measuring Add-on Startup Performance
  • Revision id: 85705
  • Created:
  • Creator: Jorge.villalobos
  • Is current revision? No
  • Comment one or more formatting changes; page display name changed to 'Measuring Add-on Startup Performance'

Revision Content

Using Talos

Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.

Prerequisites

The StandaloneTalos page explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.

Set Up

  1. You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
  2. Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).
  3. On addon.config, add any preferences and their corresponding values to the preferences variable if they are necessary to reliably test your add-on's startup performance.
  4. On addon.config, add a path to your XPI file to the extensions variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.
  5. On both files, you can remove or comment out all tests in the tests section, except for ts and ts_paint.

Running the tests

To tun the tests, close all instances of Firefox you have open.

To run the baseline tests:

python run_tests.py -s baseline.config

To run the add-on tests:

python run_tests.py -s addon.config

Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should begin similar to this:

RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58

Interpreting the results

The first set of 20 measurements are the ones that correspond to startup time. This is measured from the time the application process is started until a web page is loaded.

An average of the test results (discarding the highest one) is included at the end:

RETURN: ts: 604.58

The overhead percentage as shown on AMO pages is calculated with the formula:

overhead_percentage = (addon_average - baseline_average) / baseline_average

It represents the average add-on startup overhead as a percentage of the baseline average startup time.

These are warm startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and other files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a cold startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it would be a more realistic measurement of add-on performance impact. However, testing cold startup is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.

It's also important to keep in mind that the test results are very dependent on what you're running on your system simultaneously. For more consistent results you should run these tests will no other applications open.

Using about:startup

Revision Source

<h2>Using Talos</h2>
<p><a class=" link-https" href="https://wiki.mozilla.org/Buildbot/Talos" title="https://wiki.mozilla.org/Buildbot/Talos">Talos</a> is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight <a class=" link-https" href="https://addons.mozilla.org/en-US/firefox/performance/" title="https://addons.mozilla.org/en-US/firefox/performance/">add-ons with high startup overhead</a>.</p>
<h3>Prerequisites</h3>
<p>The <a class=" link-https" href="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home" title="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home">StandaloneTalos page</a> explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.</p>
<h3>Set Up</h3>
<ol> <li>You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.</li> <li>Make sure both files point to the right path of the Firefox executable in the <em>browser_path</em> variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).</li> <li>On addon.config, add any preferences and their corresponding values to the <em>preferences</em> variable if they are necessary to reliably test your add-on's startup performance.</li> <li>On addon.config, add a path to your XPI file to the <em>extensions</em> variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.</li> <li>On both files, you can remove or comment out all tests in the <em>tests</em> section, except for <em>ts</em> and <em>ts_paint</em>.</li>
</ol>
<h3>Running the tests</h3>
<p>To tun the tests, close all instances of Firefox you have open.</p>
<p>To run the baseline tests:</p>
<p><code>python run_tests.py -s baseline.config</code></p>
<p>To run the add-on tests:</p>
<p><code>python run_tests.py -s addon.config</code></p>
<p>Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should begin similar to this:</p>
<pre class="eval">RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58
</pre>
<h3>Interpreting the results</h3>
<p>The first set of 20 measurements are the ones that correspond to startup time. This is measured from the time the application process is started until a web page is loaded.</p>
<p>An average of the test results (discarding the highest one) is included at the end:</p>
<p><code>RETURN: ts: 604.58</code></p>
<p>The overhead percentage as shown on AMO pages is calculated with the formula:</p>
<p><code>overhead_percentage = (addon_average - baseline_average) / baseline_average</code></p>
<p>It represents the average add-on startup overhead as a percentage of the baseline average startup time.</p>
<p>These are <em>warm</em> startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and other files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a <em>cold</em> startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it would be a more realistic measurement of add-on performance impact. However, testing cold startup is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.</p>
<p>It's also important to keep in mind that the test results are very dependent on what you're running on your system simultaneously. For more consistent results you should run these tests will no other applications open.</p>
<h2>Using about:startup</h2>
Revert to this revision