Profiling with Xperf

  • Revision slug: Performance/Profiling_with_Xperf
  • Revision title: Profiling with Xperf
  • Revision id: 83156
  • Created:
  • Creator: jlebar
  • Is current revision? No
  • Comment Formatting, trying to make the path for your own builds stick out.; one or more formatting changes

Revision Content

Xperf is part of the Microsoft Windows Performance Toolkit, and has functionality similar to that of Shark, oprofile, and (for some things) dtrace/Instruments.  For stack walking, Windows Vista or higher is required; I haven't tested it at all on XP.

This page applies to xperf version 4.8.7701 or newer.  To see your xperf version, either run 'xperf' on a command line with no arguments, or start 'xperfview' and look at Help -> About Performance Analyzer.  (Note that it's not the first version number in the About window; that's the Windows version.)

If you have an older version, you will experience bugs, especially around symbol loading for local builds. 

Installation

For all versions, the tools are part of the latest Windows 7 SDK (SDK Version 7.1).  Use the web installer to install at least the "Win32 Development Tools".  Once the SDK installed, execute either wpt_x86.msi or wpt_x64.msi in the Redist/Windows Performance Toolkit folder of the SDK's install location (typically Program Files/Microsoft SDKs/Windows/v7.1/Redist/Windows Performance Toolkit) to actually install the Windows Performance Toolkit tools.

For 64-bit Windows 7 or Vista, you'll need to do a registry tweak to enable stack walking:

REG ADD "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG_DWORD -f

Symbol Server Setup

With the latest versions of the Windows Performance Toolkit, you can modify the symbol path directly from within the program via the Trace menu.  Just make sure you set the symbol paths before enabling "Load Symbols" and before opening a summary view. You can also modify the _NT_SYMBOL_PATH and _NT_SYMCACHE_PATH environment variables to make these changes permanent. 

The standard symbol path that includes both Mozilla's and Microsoft's symbol server configuration is as follows:

_NT_SYMCACHE_PATH: C:\symbols
_NT_SYMBOL_PATH: srv*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://symbols.mozilla.org/firefox/

To add symbols from your own builds, add C:\path\to\objdir\dist\bin to _NT_SYMBOL_PATH.  As with all Windows paths, the symbol path uses semicolons (;) as separators.

Make sure you select the Trace -> Load Symbols menu option in the Windows Performance Analyzer (xperfview).

There seems to be a bug in xperf and symbols; it is very sensitive to when the symbol path is edited.  If you change it within the program, you'll have to close all summary tables and reopen them for it to pick up the new symbol path data.
You'll have to agree to a EULA for the Microsoft symbols -- if you're not prompted for this, then something isn't configured right in your symbol path.  (Again, make sure that the directories exist; if they don't, it's a silent error.)

Quick Start

All these tools will live, by default, in C:\Program Files\Microsoft Windows Performance Toolkit.  Either run these commands from there, or add the directory to your path. You will need to use an elevated command prompt to start or stop profiling.

Start recording data:

xperf -on latency -stackwalk profile

"Latency" is a special provider name that turns on a few predefined kernel providers; run "xperf -providers k" to view a full list of providers and groups.  You can combine providers, e.g. "xperf -on DiagEasy+FILE_IO".  "-stackwalk profile" tells xperf to capture a stack for each PROFILE event; you could also do "-stackwalk profile+file_io" to capture a stack on each cpu profile tick and each file io completion event.

Stop:

xperf -d out.etl

View:

xperfview out.etl

The MSDN "Quickstart" page goes over this in more detail, as well as has good explanations of how to use xperfview.  I'm not going to repeat it here, because I'd be using essentially the same screenshots, so go look there.

Heap Profiling

xperf has good tools for heap allocation profiling, but they have one major limitation: you can't build with jemalloc and get heap events generated.  The stock windows CRT allocator is horrible about fragmentation, and causes memory usage to rise drastically even if only a small fraction of that memory is in use.  However, even despite this, it's a useful way to track allocations/deallocations.

Capturing Heap Data

The "-heap" option is used to set up heap tracing.  Firefox generates lots of events, so you may want to play with the BufferSize/MinBuffers/MaxBuffers options as well to ensure that you don't get dropped events.  Also, when recording the stack, I've found that a heap trace is often missing module information (I believe this is a bug in xperf).  It's possible to get around that by doing a simultaneous capture of non-heap data.

To start a trace session, launching a new Firefox instance:

xperf -on base
xperf -start heapsession -heap -PidNewProcess "./firefox.exe -P test -no-remote" -stackwalk HeapAlloc+HeapRealloc -BufferSize 512 -MinBuffers 128 -MaxBuffers 512

To stop a session and merge the resulting files:

xperf -stop heapsession -d heap.etl
xperf -d main.etl
xperf -merge main.etl heap.etl result.etl

"result.etl" will contain your merged data; you can delete main.etl and heap.etl.  Note that it's possible to capture even more data for the non-heap profile; for example, you might want to be able to correlate heap events with performance data, so you can do "xperf -on base -stackwalk profile".

In the viewer, when summary data is viewed for heap events (Heap Allocations Outstanding, etc. all lead to the same summary graphs), 3 types of allocations are listed -- AIFI , AIFO, AOFI.  This is shorthand for "Allocated Inside, Freed Inside", "Allocated Inside, Freed Outside", "Allocated Outside, Freed Inside".  These refer to the time range that was selected for the summary graph; for example, something that's in the AOFI category was allocated before the start of the selected time range, but the free event happened inside.

Tips

  • In the summary views, the yellow bar can be dragged left and right to change the grouping -- for example, drag it to the left of the Module column to have grouping happen only by process (stuff that's to the left), so that you get symbols in order of weight, regardless of what module they're in.
  • Dragging the columns around will change grouping in various ways; experiment to get the data that you're looking for.  Also experiment with turning columns on and off; removing a column will allow data to be aggregated without considering that column's contributions.

Building Firefox

To get good data from a Firefox build it is important to build with the following options in your mozconfig.

export CFLAGS="-Oy-"
export CXXFLAGS="-Oy-"

This disables frame-pointer optimization which lets xperf do a much better job unwinding the stack.  Traces can be captured fine without this option (for example, from nightlies), but the stack information will not be useful.

ac_add_options --enable-debugger-info-modules=yes

This gives us symbols.

For More Information

Microsoft's documentation for xperf is pretty good; there is a lot of depth to this tool, and you should look there for more details.

Revision Source

<p>Xperf is part of the Microsoft Windows Performance Toolkit, and has functionality similar to that of Shark, oprofile, and (for some things) dtrace/Instruments.  For stack walking, Windows Vista or higher is required; I haven't tested it at all on XP.</p>
<div class="warning">
<p>This page applies to xperf version <strong>4.8.7701 or newer</strong>.  To see your xperf version, either run '<code>xperf</code>' on a command line with no arguments, or start '<code>xperfview</code>' and look at Help -&gt; About Performance Analyzer.  (Note that it's not the first version number in the About window; that's the Windows version.)</p>
<p>If you have an older version, you will experience bugs, especially around symbol loading for local builds.  </p>
</div>
<h3>Installation</h3>
<p>For all versions, the tools are part of the latest <a class=" external" href="http://www.microsoft.com/downloads/details.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b&amp;displaylang=en" title="http://www.microsoft.com/downloads/details.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b&amp;displaylang=en">Windows 7 SDK (SDK Version 7.1)</a>.  Use the web installer to install at least the "Win32 Development Tools".  Once the SDK installed, execute either <code>wpt_x86.msi</code> or <code>wpt_x64.msi</code> in the <code>Redist/Windows Performance Toolkit </code>folder of the SDK's install location (typically Program Files/Microsoft SDKs/Windows/v7.1/Redist/Windows Performance Toolkit) to actually install the Windows Performance Toolkit tools.</p>
<div class="note">For 64-bit Windows 7 or Vista, you'll need to do a registry tweak to enable stack walking:<br>
<br>
<code>REG ADD "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG_DWORD -f</code></div>
<h3>Symbol Server Setup</h3>
<p>With the latest versions of the Windows Performance Toolkit, you can modify the symbol path directly from within the program via the Trace menu.  Just make sure you set the symbol paths before enabling "Load Symbols" and before opening a summary view. You can also modify the <code>_NT_SYMBOL_PATH</code> and <code>_NT_SYMCACHE_PATH</code> environment variables to make these changes permanent. </p> <p>The standard symbol path that includes both Mozilla's and Microsoft's symbol server configuration is as follows:</p>
<p><code>_NT_SYMCACHE_PATH: C:\symbols<br>
_NT_SYMBOL_PATH: <span class="nowiki">srv*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://symbols.mozilla.org/firefox/</span></code></p>
<p>To add symbols <strong>from your own builds</strong>, add <code>C:\path\to\objdir\dist\bin</code> to <code>_NT_SYMBOL_PATH</code>.  As with all Windows paths, the symbol path uses semicolons (<code>;</code>) as separators.</p>
<p>Make sure you select the Trace -&gt; Load Symbols menu option in the Windows Performance Analyzer (xperfview).</p>
<div class="note">There seems to be a bug in xperf and symbols; it is very sensitive to when the symbol path is edited.  If you change it within the program, you'll have to close all summary tables and reopen them for it to pick up the new symbol path data.</div>
<div class="note">You'll have to agree to a EULA for the Microsoft symbols -- if you're not prompted for this, then something isn't configured right in your symbol path.  (Again, make sure that the directories exist; if they don't, it's a silent error.)</div><h3>Quick Start</h3>
<p>All these tools will live, by default, in C:\Program Files\Microsoft Windows Performance Toolkit.  Either run these commands from there, or add the directory to your path. You will need to use an elevated command prompt to start or stop profiling.</p>
<p>Start recording data:</p>
<p><code>xperf -on latency -stackwalk profile</code></p>
<p>"Latency" is a special provider name that turns on a few predefined kernel providers; run "xperf -providers k" to view a full list of providers and groups.  You can combine providers, e.g. "xperf -on DiagEasy+FILE_IO".  "-stackwalk profile" tells xperf to capture a stack for each PROFILE event; you could also do "-stackwalk profile+file_io" to capture a stack on each cpu profile tick and each file io completion event.</p>
<p>Stop:</p>
<p><code>xperf -d out.etl</code></p>
<p>View:</p>
<p><code>xperfview out.etl</code></p>
<p>The MSDN "<a class="external" href="http://msdn.microsoft.com/en-us/library/ff190971%28v=VS.85%29.aspx">Quickstart</a>" page goes over this in more detail, as well as has good explanations of how to use xperfview.  I'm not going to repeat it here, because I'd be using essentially the same screenshots, so go look there.</p>
<h3>Heap Profiling</h3>
<p>xperf has good tools for heap allocation profiling, but they have one major limitation: you can't build with jemalloc and get heap events generated.  The stock windows CRT allocator is horrible about fragmentation, and causes memory usage to rise drastically even if only a small fraction of that memory is in use.  However, even despite this, it's a useful way to track allocations/deallocations.</p>
<h4>Capturing Heap Data</h4>
<p>The "-heap" option is used to set up heap tracing.  Firefox generates lots of events, so you may want to play with the BufferSize/MinBuffers/MaxBuffers options as well to ensure that you don't get dropped events.  Also, when recording the stack, I've found that a heap trace is often missing module information (I believe this is a bug in xperf).  It's possible to get around that by doing a simultaneous capture of non-heap data.</p>
<p>To start a trace session, launching a new Firefox instance:</p>
<p><code>xperf -on base<br>
xperf -start heapsession -heap -PidNewProcess "./firefox.exe -P test -no-remote" -stackwalk HeapAlloc+HeapRealloc -BufferSize 512 -MinBuffers 128 -MaxBuffers 512</code></p>
<p>To stop a session and merge the resulting files:</p>
<p><code>xperf -stop heapsession -d heap.etl<br>
xperf -d main.etl<br>
xperf -merge main.etl heap.etl result.etl</code></p>
<p>"result.etl" will contain your merged data; you can delete main.etl and heap.etl.  Note that it's possible to capture even more data for the non-heap profile; for example, you might want to be able to correlate heap events with performance data, so you can do "<code>xperf -on base -stackwalk profile</code>".</p>
<p>In the viewer, when summary data is viewed for heap events (Heap Allocations Outstanding, etc. all lead to the same summary graphs), 3 types of allocations are listed -- AIFI , AIFO, AOFI.  This is shorthand for "Allocated Inside, Freed Inside", "Allocated Inside, Freed Outside", "Allocated Outside, Freed Inside".  These refer to the time range that was selected for the summary graph; for example, something that's in the AOFI category was allocated before the start of the selected time range, but the free event happened inside.</p>
<h3>Tips</h3>
<ul> <li>In the summary views, the yellow bar can be dragged left and right to change the grouping -- for example, drag it to the left of the Module column to have grouping happen only by process (stuff that's to the left), so that you get symbols in order of weight, regardless of what module they're in.</li> <li>Dragging the columns around will change grouping in various ways; experiment to get the data that you're looking for.  Also experiment with turning columns on and off; removing a column will allow data to be aggregated without considering that column's contributions.</li>
</ul>
<h3>Building Firefox</h3>
<p>To get good data from a Firefox build it is important to build with the following options in your mozconfig.</p>
<p><code>export CFLAGS="-Oy-"<br>
export CXXFLAGS="-Oy-"</code></p>
<p>This disables frame-pointer optimization which lets xperf do a much better job unwinding the stack.  Traces can be captured fine without this option (for example, from nightlies), but the stack information will not be useful.</p>
<p><code>ac_add_options --enable-debugger-info-modules=yes</code></p>
<p>This gives us symbols.</p>
<h3>For More Information</h3>
<p>Microsoft's <a class="external" href="http://msdn.microsoft.com/en-us/library/ff191077.aspx" title="http://msdn.microsoft.com/en-us/library/ff191077.aspx">documentation for xperf</a> is pretty good; there is a lot of depth to this tool, and you should look there for more details.</p>
Revert to this revision