MDN may have intermittent access issues April 18 13:00 - April 19 01:00 UTC. See whistlepig.mozilla.org for all notifications.

mozilla

Revision 85702 of Measuring Add-on Startup Performance

  • Revision slug: User:Jorge.villalobos/Testing_Add-on_Startup_Performance
  • Revision title: Measuring Add-on Startup Performance
  • Revision id: 85702
  • Created:
  • Creator: Jorge.villalobos
  • Is current revision? No
  • Comment 1 words added, 67 words removed

Revision Content

Using Talos

Talos is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight add-ons with high startup overhead.

Prerequisites

The StandaloneTalos page explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.

Set Up

  1. You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.
  2. Make sure both files point to the right path of the Firefox executable in the browser_path variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).
  3. On addon.config, add any preferences and their corresponding values to the preferences variable if they are necessary to reliably test your add-on's startup performance.
  4. On addon.config, add a path to your XPI file to the extensions variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.
  5. On both files, you can remove or comment out all tests in the tests section, except for ts and ts_paint.

Running the tests

To tun the tests, close all instances of Firefox you have open.

To run the baseline tests:

python run_tests.py -s baseline.config

To run the add-on tests:

python run_tests.py -s addon.config

Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should begin similar to this:

RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58

Interpreting the results

The first set of 20 measurements are the ones that correspond to startup time. An average (after discarding the highest test result) is included at the end:

RETURN: ts: 604.58

The overhead percentage as shown on AMO pages is calulated with the formula:

overhead_percentage = (addon_average - baseline_average) / baseline_average

It represents the average add-on startup overhead as a percentage of the baseline average startup time.

Using about:startup

Revision Source

<h2>Using Talos</h2>
<p><a class=" link-https" href="https://wiki.mozilla.org/Buildbot/Talos" title="https://wiki.mozilla.org/Buildbot/Talos">Talos</a> is a performance testing framework written in Python. It is used to test Firefox performance in general, and has been adapted to test add-on startup performance as well. The Talos system is currently in use on AMO to calculate the startup performance of add-ons hosted there and highlight <a class=" link-https" href="https://addons.mozilla.org/en-US/firefox/performance/" title="https://addons.mozilla.org/en-US/firefox/performance/">add-ons with high startup overhead</a>.</p>
<h3>Prerequisites</h3>
<p>The <a class=" link-https" href="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home" title="https://wiki.mozilla.org/StandaloneTalos#How_to_set_up_Talos_for_testing_at_home">StandaloneTalos page</a> explains how to download and get Talos up and running. It is not necessary to download or install a nightly build of Firefox if you're planning on testing your add-on's startup performance only on stable Firefox builds.</p>
<h3>Set Up</h3>
<ol> <li>You'll need 2 config files: one for your baseline numbers with no add-ons installed, and one to run with your add-on. Create a copy of addon.config (in the Talos dir) and name it baseline.config.</li> <li>Make sure both files point to the right path of the Firefox executable in the <em>browser_path</em> variable. On Mac OS you need to point to the internal binary (/Applications/Firefox.app/Contents/MacOS/firefox-bin).</li> <li>On addon.config, add any preferences and their corresponding values to the <em>preferences</em> variable if they are necessary to reliably test your add-on's startup performance.</li> <li>On addon.config, add a path to your XPI file to the <em>extensions</em> variable. If your add-on needs to be unzipped in the profile folder in order to work, then the path should point to the directory of the unzipped package.</li> <li>On both files, you can remove or comment out all tests in the <em>tests</em> section, except for <em>ts</em> and <em>ts_paint</em>.</li>
</ol>
<h3>Running the tests</h3>
<p>To tun the tests, close all instances of Firefox you have open.</p>
<p>To run the baseline tests:</p>
<p><code>python run_tests.py -s baseline.config</code></p>
<p>To run the add-on tests:</p>
<p><code>python run_tests.py -s addon.config</code></p>
<p>Talos repeatedly opens and closes Firefox to run these tests. They take a few minutes to complete. The output should begin similar to this:</p>
<pre class="eval">RETURN:s: firefox_testing
firefox_testing: 
		Started Tue, 03 May 2011 12:26:31
Running test ts: 
		Started Tue, 03 May 2011 12:26:31
	Screen width/height:1920/1200
	colorDepth:24
	Browser inner width/height: 1024/653

i,val
0,740
1,612
2,615
3,583
4,609
5,594
6,594
7,619
8,581
9,607
10,602
11,589
12,592
13,612
14,587
15,588
16,599
17,578
18,760
19,586
RETURN: ts: 604.58
</pre><h3>Interpreting the results</h3>
<p>The first set of 20 measurements are the ones that correspond to startup time. An average (after discarding the highest test result) is included at the end:</p>
<p><code>RETURN: ts: 604.58</code></p>
<p>The overhead percentage as shown on AMO pages is calulated with the formula:</p>
<p><code>overhead_percentage = (addon_average - baseline_average) / baseline_average</code></p>
<p>It represents the average add-on startup overhead as a percentage of the baseline average startup time.</p>
<h2>Using about:startup</h2>
Revert to this revision