Crash Data Analysis

  • Revision slug: Crash_Data_Analysis
  • Revision title: Crash Data Analysis
  • Revision id: 376339
  • Created:
  • Creator: Benjamin Smedberg
  • Is current revision? No
  • Comment

Revision Content

The Mozilla crash reporting system is a treasure trove of data. Mozilla uses this data to measure overall product stability, prioritize and diagnose specific crash issues, and verify fixes that have been made.

Crash Report Contents

A typical crash report consists of a minidump file and a collection of metadata fields.

Minidump Files

A minidump contains at least the following information:

  • Details about the exception which led to the crash
  • Information about each thread in the process: the address which was executing and the register state at the time the process stopped.
  • A list of shared libraries loaded into the process at the time of the crash
  • The stack memory of each thread.
  • The memory right around the crashing address.

A minidump may also contain other memory regions if requested by the application, or other platform-specific data.

Metadata

Along with the minidump, the crash reporting system includes metadata fields. Firefox code and extensions can add arbitrary fields at runtime, as well as append to a "notes" field. Here are some of the basics:

ProductName
crash-stats.mozilla.com collects crashes for Firefox (Desktop), Firefox for Android, Firefox OS, Thunderbird, SeaMonkey, and a few others. This is a required field.
Version
The version number of build  that was running when the crash occurred.
BuildID
The build ID of the build that was running when the crash occurred.
Vendor
"Mozilla" in every case, even for partner builds of Firefox.
CrashTime
The time when the crash occurred, as returned by the time() C function.
StartupTime
The time when Firefox started. By comparing the startup time and the crash time, we can see how long Firefox was alive before it crashed.
InstallTime
The time when this particular version of Firefox was installed by the user.
ReleaseChannel
The update channel that the user is on. Typically "nightly", "aurora", "beta", or "release", but this may also be other values like "release-cck-partner".
Theme
The currently enabled theme.
Add-ons
A list of the addons currently enabled at the time of the crash. This takes the form of "addonid:version,[addonid:version...]". This value could be empty if the crash happens during startup before the addon manager is enabled, and on products/platforms which don't support addons.
id
The application "ID". This is a string like "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}" for Firefox which addons use to match compatibility.
Notes
Freetext notes added by application and extension code to the crash report.
AdapterVendorID/AdapterDeviceID
When present, indicates the PCI vendor and device ID of the graphics hardware.
Winsock_LSP
On Windows, a string of data from the Windows OS about the list of installed LSPs (Layered Service Provider).
FramePoisonSize/FramePoisonBase
Information about the address and size of the "frame poisoning region" used to mitigate errors in the layout code. See roc's blog for details on frame poisoning.
URL
The website which the user most recently visited in the browser before the crash. Users have the option of opting in or out of sending the current URL. This information may not always be valuable, because pages in other tabs or windows may be responsible for a particular crash. This value is considered personally-identifying information (PII) and is only available to Mozilla employees.
Throttleable
On the Firefox release channel, we typically process only 10% of crashes. If Throttleable=0, then this report is being submitted from the about:crashes UI and should always be processed.
SecondsSinceLastCrash
...
TotalVirtualMemory
SystemMemoryUsePercentage
AvailableVirtualMemory
AvailablePageFile
AvailablePhysicalMemory
Statistics about memory information (Windows-only?).
ProcessType
When the main Firefox process crashes, this will not be present. But when a plugin or content process crashes, this will be "plugin" or "content".
PluginFilename
PluginName
PluginVersion
PluginCpuUsage
When a plugin process crashes, details about the plugin loaded into that process.
NumberOfProcessors
...
Email
Users may opt in to providing their email address so that Mozilla may contact them about their crash report.
OOMAllocationSize
When Firefox intentionally aborts because an allocation fails, this annotation will indicate the size of the attempted allocation.
IsGarbageCollecting
...

Crash Report Processing

After a crash is submitted, the Socorro system gives it a unique ID and stores it for processing. It uses the breakpad crash processor to translate the minidump into a stack trace for the crash. Then it attempts to build a "signature" for the crash based on the functions near the top of the stack. For more information about the signature generation, see this very old wiki page.

Reports and Queries

The most common way to access crash report data is via the website crash-stats.mozilla.com. This websites has built-in reports of "topcrashes" for each release grouped by signature. There is also a custom query tool which allows users to limit searches on more precise information.

For more automated usage, a summary of each day's crash reports is published as a CSV file, as well as batch analysis jobs. These can be found at https://crash-analysis.mozilla.com/crash_analysis/

Finally, a set of Mozilla employees have access to directly query the underlying data in either SQL summary or using mapreduce on the storage cluster. If you are interested in obtaining this advanced access, contact Benjamin Smedberg.

Revision Source

<p>The Mozilla crash reporting system is a treasure trove of data. Mozilla uses this data to measure overall product stability, prioritize and diagnose specific crash issues, and verify fixes that have been made.</p>
<h2 id="Crash_Report_Contents">Crash Report Contents</h2>
<p>A typical crash report consists of a minidump file and a collection of metadata fields.</p>
<h3 id="Minidump_Files">Minidump Files</h3>
<p>A <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx" title="http://msdn.microsoft.com/en-us/library/windows/desktop/ms680369%28v=vs.85%29.aspx">minidump</a> contains at least the following information:</p>
<ul>
  <li>Details about the exception which led to the crash</li>
  <li>Information about each thread in the process: the address which was executing and the register state at the time the process stopped.</li>
  <li>A list of shared libraries loaded into the process at the time of the crash</li>
  <li>The stack memory of each thread.</li>
  <li>The memory right around the crashing address.</li>
</ul>
<p>A minidump may also contain other memory regions if requested by the application, or other platform-specific data.</p>
<h3 id="Metadata">Metadata</h3>
<p>Along with the minidump, the crash reporting system includes metadata fields. Firefox code and extensions can add arbitrary fields at runtime, as well as append to a "notes" field. Here are some of the basics:</p>
<dl>
  <dt>
    ProductName</dt>
  <dd>
    crash-stats.mozilla.com collects crashes for Firefox (Desktop), Firefox for Android, Firefox OS, Thunderbird, SeaMonkey, and a few others. This is a required field.</dd>
  <dt>
    Version</dt>
  <dd>
    The version number of build&nbsp; that was running when the crash occurred.</dd>
  <dt>
    BuildID</dt>
  <dd>
    The build ID of the build that was running when the crash occurred.</dd>
  <dt>
    Vendor</dt>
  <dd>
    "Mozilla" in every case, even for partner builds of Firefox.</dd>
  <dt>
    CrashTime</dt>
  <dd>
    The time when the crash occurred, as returned by the time() C function.</dd>
  <dt>
    StartupTime</dt>
  <dd>
    The time when Firefox started. By comparing the startup time and the crash time, we can see how long Firefox was alive before it crashed.</dd>
  <dt>
    InstallTime</dt>
  <dd>
    The time when this particular version of Firefox was installed by the user.</dd>
  <dt>
    ReleaseChannel</dt>
  <dd>
    The update channel that the user is on. Typically "nightly", "aurora", "beta", or "release", but this may also be other values like "release-cck-partner".</dd>
  <dt>
    Theme</dt>
  <dd>
    The currently enabled theme.</dd>
  <dt>
    Add-ons</dt>
  <dd>
    A list of the addons currently enabled at the time of the crash. This takes the form of "addonid:version,[addonid:version...]". This value could be empty if the crash happens during startup before the addon manager is enabled, and on products/platforms which don't support addons.</dd>
  <dt>
    id</dt>
  <dd>
    The application "ID". This is a string like "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}" for Firefox which addons use to match compatibility.</dd>
  <dt>
    Notes</dt>
  <dd>
    Freetext notes added by application and extension code to the crash report.</dd>
  <dt>
    AdapterVendorID/AdapterDeviceID</dt>
  <dd>
    When present, indicates the PCI vendor and device ID of the graphics hardware.</dd>
  <dt>
    Winsock_LSP</dt>
  <dd>
    On Windows, a string of data from the Windows OS about the list of installed LSPs (Layered Service Provider).</dd>
  <dt>
    FramePoisonSize/FramePoisonBase</dt>
  <dd>
    Information about the address and size of the "frame poisoning region" used to mitigate errors in the layout code. See <a href="http://robert.ocallahan.org/2010/10/mitigating-dangling-pointer-bugs-using_15.html" title="http://robert.ocallahan.org/2010/10/mitigating-dangling-pointer-bugs-using_15.html">roc's blog</a> for details on frame poisoning.</dd>
  <dt>
    URL</dt>
  <dd>
    The website which the user most recently visited in the browser before the crash. Users have the option of opting in or out of sending the current URL. This information may not always be valuable, because pages in other tabs or windows may be responsible for a particular crash. This value is considered personally-identifying information (PII) and is only available to Mozilla employees.</dd>
  <dt>
    Throttleable</dt>
  <dd>
    On the Firefox release channel, we typically process only 10% of crashes. If Throttleable=0, then this report is being submitted from the about:crashes UI and should always be processed.</dd>
  <dt>
    SecondsSinceLastCrash</dt>
  <dd>
    ...</dd>
  <dt>
    TotalVirtualMemory</dt>
  <dt>
    SystemMemoryUsePercentage</dt>
  <dt>
    AvailableVirtualMemory</dt>
  <dt>
    AvailablePageFile</dt>
  <dt>
    AvailablePhysicalMemory</dt>
  <dd>
    Statistics about memory information (Windows-only?).</dd>
  <dt>
    ProcessType</dt>
  <dd>
    When the main Firefox process crashes, this will not be present. But when a plugin or content process crashes, this will be "plugin" or "content".</dd>
  <dt>
    PluginFilename</dt>
  <dt>
    PluginName</dt>
  <dt>
    PluginVersion</dt>
  <dt>
    PluginCpuUsage</dt>
  <dd>
    When a plugin process crashes, details about the plugin loaded into that process.</dd>
  <dt>
    NumberOfProcessors</dt>
  <dd>
    ...</dd>
  <dt>
    Email</dt>
  <dd>
    Users may opt in to providing their email address so that Mozilla may contact them about their crash report.</dd>
  <dt>
    OOMAllocationSize</dt>
  <dd>
    When Firefox intentionally aborts because an allocation fails, this annotation will indicate the size of the attempted allocation.</dd>
  <dt>
    IsGarbageCollecting</dt>
  <dd>
    ...</dd>
</dl>
<h2>Crash Report Processing<span id="cke_bm_93C" style="display: none;"> </span></h2>
<p>After a crash is submitted, the <a href="https://github.com/mozilla/socorro" title="https://github.com/mozilla/socorro">Socorro</a> system gives it a unique ID and stores it for processing. It uses the <a href="http://code.google.com/p/google-breakpad/" title="http://code.google.com/p/google-breakpad/">breakpad</a> crash processor to translate the minidump into a stack trace for the crash. Then it attempts to build a "signature" for the crash based on the functions near the top of the stack. For more information about the signature generation, see <a href="http://code.google.com/p/socorro/wiki/SignatureGeneration" title="http://code.google.com/p/socorro/wiki/SignatureGeneration">this very old wiki page</a>.</p>
<h2>Reports and Queries</h2>
<p>The most common way to access crash report data is via the website <a href="https://crash-stats.mozilla.com/" title="https://crash-stats.mozilla.com/">crash-stats.mozilla.com</a>. This websites has built-in reports of "topcrashes" for each release grouped by signature. There is also a custom query tool which allows users to limit searches on more precise information.</p>
<p>For more automated usage, a summary of each day's crash reports is published as a CSV file, as well as batch analysis jobs. These can be found at https://crash-analysis.mozilla.com/crash_analysis/</p>
<p>Finally, a set of Mozilla employees have access to directly query the underlying data in either SQL summary or using mapreduce on the storage cluster. If you are interested in obtaining this advanced access, contact <a href="mailto:benjamin@smedbergs.us?subject=crash-stats%20access%20query" title="mailto:benjamin@smedbergs.us?subject=crash-stats%20access%20query">Benjamin Smedberg</a>.</p>
Revert to this revision