mozilla

Revision 85677 of Measuring add-on startup performance

  • Revision slug: Performance/Measuring_add-on_startup_performance
  • Revision title: Measuring add-on startup performance
  • Revision id: 85677
  • Created:
  • Creator: Jorge.villalobos
  • Is current revision? No
  • Comment First draft, which was accepted on bug 650778; page created, 850 words added

Revision Content

Mozilla kicked off an Add-on Performance initiative that intends to make users more aware of the performance impact of the add-ons they install. This article explains how to measure add-on startup time and produce similar results as the ones on AMO.

The Performance Best Practices documentation explains how to make add-ons that have a small footprint.

Using Talos

Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.

Prerequisites

The Standalone Talos page details how to download and get Talos up and running. It's not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.

Set Up

  1. You'll need 2 config files: one for your baseline numbers (no add-ons installed), and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
  2. Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (e.g. /Applications/Firefox.app/Contents/MacOS/firefox-bin).
  3. On addon.config, add any preferences and their corresponding values to the preferences section if they are necessary to reliably test your add-on's startup performance.
  4. On addon.config, set the extensions variable to [ '/path/to/xpi' ].
  5. On both files, you should remove or comment out all tests in the tests section, except for ts.

Running the tests

To tun the tests, close all Firefox instances you have open.

To run the baseline tests:

python run_tests.py -s baseline.config

To run the add-on tests:

python run_tests.py -s addon.config

Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete.

The output should begin similar to this:

RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58

Interpreting the results

The first set of 20 measurements corresponds to startup time. This is measured from the time the application process is started until a web page is loaded.

An average of the test results (discarding the highest one) is included at the end:

RETURN: ts: 604.58

The overhead percentage as shown on AMO pages is calculated with the formula:

overhead_percentage = (addon_average - baseline_average) / baseline_average

It represents the average add-on startup overhead as a percentage of the baseline average startup time.

These are warm startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a cold startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it is a more realistic measurement of add-on startup performance overhead. However, cold startup testing is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.

It's also important to keep in mind that the test results are very dependent on your system and what you're running on it. For more consistent results you should run these tests wiith no other applications open.

Using About Startup

The About Startup add-on provides a much simpler way to measure startup performance. Once installed, it will gather startup data and display it in the about:startup page. The add-on is very lightweight, so it shouldn't have a noticeable negative effect on your baseline and add-on tests.

To test with About Startup, you should set up a new Firefox profile, install this add-on, and then open and quit Firefox repeatedly. If you want to match Talos, you should do this 20 times. Then look at the results in the about:startup page. This should provide your baseline numbers. Then repeat the process with both About Startup and your add-on installed. The overhead can be calculated using the same formula.

The about:startup page shows 3 different time measurements: main, sessionRestored and firstPaint. main happens very early in the startup process and should not be affected by add-ons being installed. The firstPaint time should be roughly similar to what Talos tests. This can vary between systems, though. sessionRestored is fired later in the process, when Firefox has loaded all tabs from the saved session.

Revision Source

<p>Mozilla kicked off an <a class=" external" href="http://blog.mozilla.com/addons/2011/04/01/improving-add-on-performance/" title="http://blog.mozilla.com/addons/2011/04/01/improving-add-on-performance/">Add-on Performance</a> initiative that intends to make users more aware of the performance impact of the add-ons they install. This article explains how to measure add-on startup time and produce similar results as the ones on AMO.</p>
<p>The <a href="/en/Extensions/Performance_best_practices_in_extensions" title="en/Extensions/Performance best practices in extensions">Performance Best Practices</a> documentation explains how to make add-ons that have a small footprint.</p>
<h2>Using Talos</h2>
<p><a class=" link-https" href="https://wiki.mozilla.org/Buildbot/Talos" title="https://wiki.mozilla.org/Buildbot/Talos">Talos</a> is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight <a class=" link-https" href="https://addons.mozilla.org/en-US/firefox/performance/" title="https://addons.mozilla.org/en-US/firefox/performance/">add-ons with high startup overhead</a>.</p>
<h3>Prerequisites</h3>
<p>The <a class=" link-https" href="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home" title="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home">Standalone Talos page</a> details how to download and get Talos up and running. It's not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.</p>
<h3>Set Up</h3>
<ol> <li>You'll need 2 config files: one for your baseline numbers (no add-ons installed), and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.</li> <li>Make sure both files point to the right path of the Firefox executable in the <em>browser_path</em> variable. On Mac OS you need to point to the internal binary (e.g. <em>/Applications/Firefox.app/Contents/MacOS/firefox-bin</em>).</li> <li>On addon.config, add any preferences and their corresponding values to the <em>preferences</em> section if they are necessary to reliably test your add-on's startup performance.</li> <li>On addon.config, set the <em>extensions</em> variable to [ '/path/to/xpi' ].</li> <li>On both files, you should remove or comment out all tests in the <em>tests</em> section, except for <em>ts</em>.</li>
</ol>
<h3>Running the tests</h3>
<p>To tun the tests, close all Firefox instances you have open.</p>
<p>To run the baseline tests:</p>
<p><code>python run_tests.py -s baseline.config</code></p>
<p>To run the add-on tests:</p>
<p><code>python run_tests.py -s addon.config</code></p>
<p>Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete.</p>
<p>The output should begin similar to this:</p>
<pre class="eval">RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58
</pre>
<h3>Interpreting the results</h3>
<p>The first set of 20 measurements corresponds to startup time. This is measured from the time the application process is started until a web page is loaded.</p>
<p>An average of the test results (discarding the highest one) is included at the end:</p>
<p><code>RETURN: ts: 604.58</code></p>
<p>The overhead percentage as shown on AMO pages is calculated with the formula:</p>
<p><code>overhead_percentage = (addon_average - baseline_average) / baseline_average</code></p>
<p>It represents the average add-on startup overhead as a percentage of the baseline average startup time.</p>
<p>These are <em>warm</em> startup tests, meaning that Firefox starts quickly because the operating system has cached and preloaded a series of libraries and files from previous Firefox startups. When Firefox loads for the first time after a system reboot, it performs a <em>cold</em> startup, which normally takes longer. In cold startup tests, the overhead of an add-on is much lower in relation to overall startup time. Cold startup is what most users experience when opening Firefox, and it is a more realistic measurement of add-on startup performance overhead. However, cold startup testing is much more difficult to set up in an automated system, so the tests performed on AMO are currently limited to warm startup.</p>
<p>It's also important to keep in mind that the test results are very dependent on your system and what you're running on it. For more consistent results you should run these tests wiith no other applications open.</p>
<h2>Using About Startup</h2>
<p>The <a class=" link-https" href="https://addons.mozilla.org/en-US/firefox/addon/about-startup/" title="https://addons.mozilla.org/en-US/firefox/addon/about-startup/">About Startup add-on</a> provides a much simpler way to measure startup performance. Once installed, it will gather startup data and display it in the about:startup page. The add-on is very lightweight, so it shouldn't have a noticeable negative effect on your baseline and add-on tests.</p>
<p>To test with About Startup, you should set up a new Firefox profile, install this add-on, and then open and quit Firefox repeatedly. If you want to match Talos, you should do this 20 times. Then look at the results in the about:startup page. This should provide your baseline numbers. Then repeat the process with both About Startup and your add-on installed. The overhead can be calculated using the same formula.</p>
<p>The about:startup page shows 3 different time measurements: <em>main</em>, <em>sessionRestored</em> and <em>firstPaint</em>. <em>main</em> happens very early in the startup process and should not be affected by add-ons being installed. The <em>firstPaint</em> time should be roughly similar to what Talos tests. This can vary between systems, though. <em>sessionRestored</em> is fired later in the process, when Firefox has loaded all tabs from the saved session.</p>
Revert to this revision