Profiling with the built-in profiler

  • Revision slug: Performance/Profiling_with_the_Built-in_Profiler
  • Revision title: Profiling with the built-in profiler
  • Revision id: 331741
  • Created:
  • Creator: lmorchard
  • Is current revision? No
  • Comment Added some links to the Cleopatra profile viewer UI and github repo

Revision Content

{{ gecko_minversion_header("16.0") }}

Firefox now has a built-in profiler (codename SPS). Having a profiler in the code base lets us, among other things, better measure responsiveness, charge performance costs more accurately, and run in environments and platforms in which external profilers aren't available, such as in the user's environment or on a locked Android device.

Note: While the profiler platform itself is completed, we are working on new feature requests. Watch the "Gecko Profiler" component in Bugzilla for more information.

Stackwalking vs. pseudostack

The profiler is designed to operate in two different modes: Pseudostack (the default) or stack walking.

Pseudostack

Pseudostack (the default) requires the code base to be instrumented with SAMPLE_LABEL("NAMESPACE", "NAME");'. This adds a RAII helper on the stack that will be used by the profiler when sampling. This mode works on all platforms and environments. For this to be effective, you do need to liberally use SAMPLE_LABEL throughout the code, and unfortunately it won't be able to dig into system library and drivers.

Because of the small overhead of the instrumentation, the sample label shouldn't be placed inside hot loops. A profile reporting that a large portion is spent in "Unknown" code indicates that the area being executed doesn't have sample label. As we focus on using this tool and add additional sample labels this will improve.

Stack walking

Stack walking is an optional, platform specific, feature that isn't complete yet. The goal is to provide stack walking on any platforms that support it. Having this feature will give us a detailed stack and help us analyze problems where we're spending time in drivers and system libraries. We're working on building the proper stack walking and symbolization required to make this step work, and are looking for help with this feature.

Availability

The profiler will operate using either the Pseudostack or stack walking configuration depending on your environment. See above for details on these.

  Custom Build Nightly Release (Gecko 15.0+)
Windows Stackwalking (Custom steps) Stackwalking Pseudo stack
Mac Stackwalking Stackwalking Pseudo stack
Linux Pseudo stack (Bug for stackwalking) Pseudo stack (Bug for stackwalking) Pseudo stack
Fennec Pseudo stack (Bug for stackwalking) Pseudo stack (Bug for stackwalking) Pseudo stack (19+)
B2G Pseudo stack (Bug for stackwalking) ??? None (Bug)

Running the profiler

  1. Switch to the nightly build or build with ac_add_options --enable-profiling (optimization recommended to get accurate profiling).
  2. Make sure you're using the latest version of the extension. If the addon doesn't show up hit 'View->Toolbar->Add-on bar'. Let the extension update itself. (You can find pre-release versions of the extension here).
  3. Control the profiler from the bottom right. Use Ctrl-Shift-O (Cmd-Shift-O on Mac OS X) to look at the current profile.

Profiling local Windows builds

If you built Firefox for Windows locally and you would like to use the local symbols with the profiler, you will need to run an additional tool; see Profiling with the Built-in Profiler and Local Symbols on Windows.

Profiling Firefox mobile

  1. You'll need a custom build, this build should be built with optimization, STRIP_FLAGS="--strip-debug" and ac_add_options --disable-elf-hack (until we fix {{bug(747033)}}) but should NOT be built with --enable-profiling.
  2. You'll need to have adb and arm-eabi-addr2line in your bash PATH, so use locate arm-eabi-addr2line (on linux) or mdfind name:arm-eabi-addr2line (on OS X) and stick an export to its location in ~/.bash_profile. The extension will invoke bash to use adb and addr2line.
  3. Install the latest experimental build from the RDP branch in your host machine's Firefox browser that has your phone reachable via ADB. This will add a icon in the top right of the browser.
  4. Set 'devtools.debugger.remote-enabled;true' in about:config for Fennec.
  5. Select target 'Mobile USB' and press 'connect'. The first run will take an additional 1 minute to pull the system libraries.

Profiling Boot to Gecko (with a real device)

There is now a script called profile.sh in the root of the B2G tree which simplifies most of the steps of grabbing profile information from the phone.

  1. You need to have a local build of B2G
  2. You need to have your phone plugged into your PC and have adb in your PATH.

./profile.sh start will restart b2g with profiling enabled.

./profile.sh ps will show running b2g processes and whether the profiler was enabled for those processes or not.

./profile.sh capture [pid or name] will initiate a capture. If you don't specify any arguments, then all currently running b2g processes will be captured. Otherwise the b2g process with the indicated pid or name will be captured. The profile script will pull the profile files from the phone, and add symbols.

The profile script uses some variables from the file .var.profile, which is generated by the build. These will allow the script to locate your objdir-gecko tree, the appropriate toolchain, and the out/target/product/<phone> tree to get symbols for the android libraries.

The .txt files will be renamed when pulled to the host and will have the following pattern: profile_HHMM_PID_NAME.txt (or .sym) If your capture includes multiple processes then they'll all have the same HHMM portion. The PID will be the PID of the process, and NAME will be the app name (as per the ps output)

The .sym files (which are the .txt files with symbols added) can then be uploaded to the cleopatra UI.

./profile.sh stop will kill the currently running b2g and restart it normally (i.e. profiling disabled)

Some extra commands available in the profile script (these will not be needed normally, but can be useful if you're working on the script):

./profile.sh ls will show all of the profile files stored on the phone (it looks in the /data/local/tmp directory)

./profile.sh signal [pid] triggers the profiler to store the current profile buffers to files on the phone.

./profile.sh pull pid [NAME [HHMM]] will pull the profile file for the indicated pid and rename it as mentioned above.

./profile.sh symbolicate filename will take the profile_HHMM_PID_NAME.txt file and create profile_HHMM_PID_NAME.sym which has symbols in it.

./profile.sh help will print out all of the commands currently supported by the script.

Profiling JS benchmark (xpcshell)

  1. You'll need a custom build of the xpcshell, including the following patches: 100µs sampling patch ({{ bug(807854) }}), and on Linux you will need the experimental patch to enable for stack walking ({{ bug(812946) }}).
  2. To profile the script run.js with IonMonkey (-I), type inference (-n) and JäegerMonkey (-m), you need to run the following command:
    $ xpcshell -m -I -n -e '
        const Ci = Components.interfaces;
        const Cc = Components.classes;
        var profiler = Cc["@mozilla.org/tools/profiler;1"].getService(Ci.nsIProfiler);
        profiler.StartProfiler(
          10000000 /* = profiler memory */,
          1 /* = sample rate: 100µs with patch, 1ms without */,
          ["stackwalk", "js"], 2 /* = features, and number of features. */
        );
      ' -f ./run.js -e '
        var profileObj = profiler.getProfileData();
        print(JSON.stringify(profileObj));
      ' | tail -n 1 > run.cleo
    The xpcshell output all benchmark information and on its last line it output the result of the profiling, you can filter it with tail -n 1 and redirect it to a file to prevent printing it in your shell.  The expected size of the output is around 100 of MB.
  3. To add symbols to your build, you need to call ./scripts/profile-symbolicate.py available in B2G repository. If libraries are not found, you will need to patch the script with {{ bug(812063) }}'s attachment.
    $ GECKO_OBJDIR=<objdir> PRODUCT_OUT=<objdir> TARGET_TOOLS_PREFIX= \
        ./scripts/profile-symbolicate.py -o run.symb.cleo run.cleo
  4. Clone Cleopatra and start the server with ./run_webserver.sh.
  5. Access Cleopatra from your web browser by loading the page localhost:8000, and upload run.symb.cleo to render the profile with most of the symbol information.

Contribute

Platform tracking bugs can be found in {{ bug("713227") }}. Source is located in {{ Source("tools/profiler") }}.

Profiler add-on repository can be found here: https://github.com/bgirard/Gecko-Profiler-Addon.

Cleopatra repository can be found here: https://github.com/bgirard/cleopatra.

Planned features

Here's a list of feature ideas that we haven't committed to yet; feel free to add your ideas:

  • A "Doctor" extension that would detect if the user's browser periodically hangs for more than 500 ms and provide a UI notification with some profile data. The profile data can either be reported to Mozilla to correlate problems, have built in heuristics for resolving the problem (disabling the offending add-on, plug-in tab, db caches etc...), or be used by support for diagnostics.
  • Automatically turn on the profiler a few seconds prior to triggering the hang detectors and attach a profile to the minidump.

Screenshot

3b.png

Revision Source

<p>{{ gecko_minversion_header("16.0") }}</p>
<p>Firefox now has a built-in profiler (codename SPS). Having a profiler in the code base lets us, among other things, better measure responsiveness, charge performance costs more accurately, and run in environments and platforms in which external profilers aren't available, such as in the user's environment or on a locked Android device.</p>
<div class="note">
  <strong>Note:</strong>&nbsp;While the profiler platform itself is completed, we are working on new feature requests. Watch the "Gecko Profiler" component in <a class="link-https" href="https://bugzilla.mozilla.org/" title="https://bugzilla.mozilla.org/">Bugzilla</a> for more information.</div>
<h2 id="Stackwalking_vs._pseudostack">Stackwalking vs. pseudostack</h2>
<p>The profiler is designed to operate in two different modes: Pseudostack (the default) or stack walking.</p>
<h3 id="Pseudostack">Pseudostack</h3>
<p>Pseudostack (the default) requires the code base to be instrumented with <code>SAMPLE_LABEL("NAMESPACE", "NAME");'</code>. This adds a RAII helper on the stack that will be used by the profiler when sampling. This mode works on all platforms and environments. For this to be effective, you do need to liberally use <code>SAMPLE_LABEL</code> throughout the code, and unfortunately it won't be able to dig into system library and drivers.</p>
<p>Because of the small overhead of the instrumentation, the sample label shouldn't be placed inside hot loops. A profile reporting that a large portion is spent in "Unknown" code indicates that the area being executed doesn't have sample label. As we focus on using this tool and add additional sample labels this will improve.</p>
<h3 id="Stack_walking">Stack walking</h3>
<p>Stack walking is an optional, platform specific, feature that isn't complete yet. The goal is to provide stack walking on any platforms that support it. Having this feature will give us a detailed stack and help us analyze problems where we're spending time in drivers and system libraries. We're working on building the proper stack walking and symbolization required to make this step work, and are looking for help with this feature.</p>
<h2 id="Availability">Availability</h2>
<p>The profiler will operate using either the Pseudostack or stack walking configuration depending on your environment. See above for details on these.</p>
<table border="1" cellpadding="1" cellspacing="1" class="standard-table">
  <thead>
    <tr>
      <th scope="row">&nbsp;</th>
      <th scope="col">Custom Build</th>
      <th scope="col">Nightly</th>
      <th scope="col">Release (Gecko 15.0+)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th scope="row">Windows</th>
      <td>Stackwalking (<a href="/en/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows" title="https://developer.mozilla.org/en/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows">Custom steps</a>)</td>
      <td>Stackwalking</td>
      <td>Pseudo stack</td>
    </tr>
    <tr>
      <th scope="row">Mac</th>
      <td>Stackwalking</td>
      <td>Stackwalking</td>
      <td>Pseudo stack</td>
    </tr>
    <tr>
      <th scope="row">Linux</th>
      <td>Pseudo stack (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=757186" title="https://bugzilla.mozilla.org/show_bug.cgi?id=757186">Bug</a> for stackwalking)</td>
      <td>Pseudo stack (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=757186" title="https://bugzilla.mozilla.org/show_bug.cgi?id=757186">Bug</a><span class="external"> for stackwalking</span>)</td>
      <td>Pseudo stack</td>
    </tr>
    <tr>
      <th scope="row">Fennec</th>
      <td>Pseudo stack (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=757186" title="https://bugzilla.mozilla.org/show_bug.cgi?id=757186">Bug</a> for stackwalking)</td>
      <td>Pseudo stack (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=757186" title="https://bugzilla.mozilla.org/show_bug.cgi?id=757186">Bug</a> for stackwalking)</td>
      <td>Pseudo stack (19+)</td>
    </tr>
    <tr>
      <th scope="row">B2G</th>
      <td>Pseudo stack (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=757186" title="https://bugzilla.mozilla.org/show_bug.cgi?id=757186">Bug</a> for stackwalking)</td>
      <td>???</td>
      <td>None (<a class="link-https" href="https://bugzilla.mozilla.org/show_bug.cgi?id=751034" title="https://bugzilla.mozilla.org/show_bug.cgi?id=751034">Bug</a>)</td>
    </tr>
  </tbody>
</table>
<h2 id="Running_the_profiler">Running the profiler</h2>
<ol>
  <li>Switch to the <a class="external" href="http://nightly.mozilla.org/" title="http://nightly.mozilla.org/">nightly build</a> or build with <code>ac_add_options --enable-profiling</code> (optimization recommended to get accurate profiling).</li>
  <li>Make sure you're using the <a class="link-https" href="https://addons.mozilla.org/en-us/firefox/addon/gecko-profiler/">latest version of the extension</a>. <strong>If the addon doesn't show up hit 'View-&gt;Toolbar-&gt;Add-on bar'</strong>. Let the extension update itself. (You can find pre-release versions of the extension <a href="https://github.com/bgirard/Gecko-Profiler-Addon/raw/master/geckoprofiler.xpi" title="https://github.com/bgirard/Gecko-Profiler-Addon/raw/master/geckoprofiler.xpi">here</a>).</li>
  <li>Control the profiler from the bottom right. Use Ctrl-Shift-O (Cmd-Shift-O on Mac OS X) to look at the current profile.</li>
</ol>
<h3 id="Profiling_local_Windows_builds">Profiling local Windows builds</h3>
<p>If you built Firefox for Windows <strong>locally</strong> and you would like to use the local symbols with the profiler, you will need to run an additional tool; see <a href="/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows" title="/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows">Profiling with the Built-in Profiler and Local Symbols on Windows</a>.</p>
<h3 id="Profiling_Firefox_mobile">Profiling Firefox mobile</h3>
<ol>
  <li>You'll need a custom build, this build should be built with optimization, <code>STRIP_FLAGS="--strip-debug"</code> and <code>ac_add_options --disable-elf-hack</code> (until we fix {{bug(747033)}}) but should <strong>NOT</strong> be built with <code>--enable-profiling</code>.</li>
  <li>You'll need to have <code>adb</code> and <code>arm-eabi-addr2line</code> in your bash <code>PATH</code>, so use <code>locate arm-eabi-addr2line</code> (on linux) or<code> mdfind name:arm-eabi-addr2line</code> (on OS X) and stick an export to its location in <code>~/.bash_profile</code>. The extension will invoke bash to use <code>adb</code> and <code>addr2line</code>.</li>
  <li>Install the <a href="https://github.com/bgirard/Gecko-Profiler-Addon/raw/RDP/geckoprofiler.xpi" title="https://github.com/bgirard/Gecko-Profiler-Addon/raw/RDP/geckoprofiler.xpi">latest experimental build from the RDP branch</a> in your host machine's Firefox browser that has your phone reachable via ADB. This will add a icon in the top right of the browser.</li>
  <li>Set 'devtools.debugger.remote-enabled;true' in about:config for Fennec.</li>
  <li>Select target 'Mobile USB' and press 'connect'. The first run will take an additional 1 minute to pull the system libraries.</li>
</ol>
<h3 id="Profiling_Boot_to_Gecko_(with_a_real_device)">Profiling Boot to Gecko (with a real device)</h3>
<p>There is now a script called <strong>profile.sh</strong> in the root of the B2G tree which simplifies most of the steps of grabbing profile information from the phone.</p>
<ol>
  <li>You need to have a local build of B2G</li>
  <li>You need to have your phone plugged into your PC and have adb in your PATH.</li>
</ol>
<p><strong>./profile.sh start</strong> will restart b2g with profiling enabled.</p>
<p><strong>./profile.sh ps</strong> will show running b2g processes and whether the profiler was enabled for those processes or not.</p>
<p><strong>./profile.sh capture [pid or name]</strong> will initiate a capture. If you don't specify any arguments, then all currently running b2g processes will be captured. Otherwise the b2g process with the indicated pid or name will be captured. The profile script will pull the profile files from the phone, and add symbols.</p>
<p>The profile script uses some variables from the file .var.profile, which is generated by the build. These will allow the script to locate your objdir-gecko tree, the appropriate toolchain, and the out/target/product/&lt;phone&gt; tree to get symbols for the android libraries.</p>
<p>The .txt files will be renamed when pulled to the host and will have the following pattern: profile_HHMM_PID_NAME.txt (or .sym) If your capture includes multiple processes then they'll all have the same HHMM portion. The PID will be the PID of the process, and NAME will be the app name (as per the ps output)</p>
<p>The .sym files (which are the .txt files with symbols added) can then be uploaded to <a href="http://people.mozilla.com/~bgirard/cleopatra/" title="http://people.mozilla.com/~bgirard/cleopatra/">the cleopatra UI</a>.</p>
<p><strong>./profile.sh stop</strong> will kill the currently running b2g and restart it normally (i.e. profiling disabled)</p>
<p>Some extra commands available in the profile script (these will not be needed normally, but can be useful if you're working on the script):</p>
<p><strong>./profile.sh ls</strong> will show all of the profile files stored on the phone (it looks in the /data/local/tmp directory)</p>
<p><strong>./profile.sh signal [pid]</strong> triggers the profiler to store the current profile buffers to files on the phone.</p>
<p><strong>./profile.sh pull pid [NAME [HHMM]]</strong> will pull the profile file for the indicated pid and rename it as mentioned above.</p>
<p><strong>./profile.sh symbolicate filename</strong> will take the profile_HHMM_PID_NAME.txt file and create profile_HHMM_PID_NAME.sym which has symbols in it.</p>
<p><strong>./profile.sh help</strong> will print out all of the commands currently supported by the script.</p>
<h3 id="Profiling_JS_benchmark_(xpcshell)">Profiling JS benchmark (xpcshell)</h3>
<ol>
  <li>You'll need a custom build of the xpcshell, including the following patches: 100µs sampling patch ({{ bug(807854) }}), and on Linux you will need the experimental patch to enable for stack walking ({{ bug(812946) }}).</li>
  <li>To profile the script <code>run.js</code> with IonMonkey (<code>-I</code>), type inference (<code>-n</code>) and JäegerMonkey (<code>-m</code>), you need to run the following command:
    <pre>
<code>$ xpcshell -m -I -n -e '
    const Ci = Components.interfaces;
    const Cc = Components.classes;
    var profiler = Cc["@mozilla.org/tools/profiler;1"].getService(Ci.nsIProfiler);
    profiler.StartProfiler(
      10000000 /* = profiler memory */,
      1 /* = sample rate: 100µs with patch, 1ms without */,
      ["stackwalk", "js"], 2 /* = features, and number of features. */
    );
  ' -f ./run.js -e '
    var profileObj = profiler.getProfileData();
    print(JSON.stringify(profileObj));
  ' | tail -n 1 &gt; run.cleo</code></pre>
    The xpcshell output all benchmark information and on its last line it output the result of the profiling, you can filter it with <code>tail -n 1</code> and redirect it to a file to prevent printing it in your shell.&nbsp; The expected size of the output is around 100 of MB.</li>
  <li>To add symbols to your build, you need to call <code>./scripts/profile-symbolicate.py</code> available in B2G repository. If libraries are not found, you will need to patch the script with {{ bug(812063) }}'s attachment.<br />
    <pre>
<code>$ GECKO_OBJDIR=&lt;objdir&gt; PRODUCT_OUT=&lt;objdir&gt; TARGET_TOOLS_PREFIX= \
    ./scripts/profile-symbolicate.py -o run.symb.cleo run.cleo</code></pre>
  </li>
  <li><a href="https://github.com/bgirard/cleopatra" title="https://github.com/bgirard/cleopatra">Clone Cleopatra</a> and start the server with <code>./run_webserver.sh</code>.</li>
  <li>Access Cleopatra from your web browser by loading the page <code>localhost:8000</code>, and upload <code>run.symb.cleo</code> to render the profile with most of the symbol information.</li>
</ol>
<h2 id="Contribute">Contribute</h2>
<p>Platform tracking bugs can be found in {{ bug("713227") }}. Source is located in {{ Source("tools/profiler") }}.</p>
<p>Profiler add-on repository can be found here: <a class="link-https" href="https://github.com/bgirard/Gecko-Profiler-Addon" rel="freelink">https://github.com/bgirard/Gecko-Profiler-Addon</a>.</p>
<p>Cleopatra repository can be found here: <a class="link-https" href="https://github.com/bgirard/cleopatra" rel="freelink">https://github.com/bgirard/cleopatra</a>.</p>
<h2 id="Planned_features">Planned features</h2>
<p>Here's a list of feature ideas that we haven't committed to yet; feel free to add your ideas:</p>
<ul>
  <li>A "Doctor" extension that would detect if the user's browser periodically hangs for more than 500 ms and provide a UI notification with some profile data. The profile data can either be reported to Mozilla to correlate problems, have built in heuristics for resolving the problem (disabling the offending add-on, plug-in tab, db caches etc...), or be used by support for diagnostics.</li>
  <li>Automatically turn on the profiler a few seconds prior to triggering the hang detectors and attach a profile to the minidump.</li>
</ul>
<h2 id="Screenshot">Screenshot</h2>
<p><a href="/@api/deki/files/6062/=profiler.png" title="profiler.png"><img alt="3b.png" class="internal default" src="/@api/deki/files/6191/=3b.png" style="width: 803px; height: 562px;" /></a></p>
Revert to this revision