Revision 61135 of Building with Profile-Guided Optimization

  • Revision slug: Building_with_Profile-Guided_Optimization
  • Revision title: Building with Profile-Guided Optimization
  • Revision id: 61135
  • Created:
  • Creator: ThomasR
  • Is current revision? No
  • Comment one or more formatting changes

Revision Content

Introduction

The Mozilla build system contains support for building with Profile-Guided Optimization (PGO) with GCC 4 or newer and Microsoft Visual C++ 2005 (Professional Edition) or newer. A PGO build consists of two passes: a first pass to build instrumented binaries, then a second pass to re-build optimized binaries using profile information gleaned from running the instrumented binaries. The Mozilla build system will run both passes for you, you simply need to provide a script to run the application through some profiling scenarios in between the build passes. {{ Note("The script does not need to be automated! If you are not running your builds in an automated environment, you can simply have it launch the browser, then interact with the browser to generate the profiling data and close it when you\'re done.") }}

Build Configuration

You will need to setup a mozconfig as normal with the build options you want. You will also need to specify one additional option in your mozconfig for the PGO build process, the script to run between build passes. You set this with mk_add_options PROFILE_GEN_SCRIPT like so:

 mk_add_options PROFILE_GEN_SCRIPT=/home/user/run-firefox.sh

This command will be run directly from make, so ensure that the file is executable. On Windows you will need to specify the interpreter name in front of the script, like so:

 mk_add_options PROFILE_GEN_SCRIPT="sh /home/user/run-firefox.sh"

The script does not have to be a shell script, it can be Perl or Python or anything else that you can run from the command line, just make sure to specify the right command for running it.

You can use the same script the nightly builds use with the following:

 mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'

The profile generation script

When your script is run it is passed in an environment variable, OBJDIR, containing the path to the object directory. A simple shell script for Linux that just launches the browser would look like this:

 #!/bin/sh
 export NO_EM_RESTART=1
 mkdir $OBJDIR/_profileprofile
 $OBJDIR/dist/bin/firefox -no-remote -profile $OBJDIR/_profileprofile

It simply creates an empty profile directory, then runs Firefox with that profile directory. You would have to manually run your profiling scenarios and close the browser.

Building

PGO builds require you to start your build from a different target in client.mk. Instead of the usual make -f client.mk build, you simply do make -f client.mk profiledbuild. {{ Note("PGO builds will take much longer than normal builds! With GCC, you will wind up doing two complete clobber builds. With MSVC, you will only do one full build, and then another pass of re-linking, but re-linking libxul can take over a half hour on a fast machine, and use over one gigabyte of RAM.") }}{{ Warning("Linking a PGO build is known to exhaust the 32 bit address space on Win32.  To work around this either use a 64 bit compiler or use the /3GB boottime switch to give the linker more address space.  See Bug 543034 for details.") }}

Revision Source

<h2 id="Introduction" name="Introduction">Introduction</h2>
<p>The Mozilla build system contains support for building with Profile-Guided Optimization (PGO) with GCC 4 or newer and Microsoft Visual C++ 2005 (Professional Edition) or newer. A PGO build consists of two passes: a first pass to build instrumented binaries, then a second pass to re-build optimized binaries using profile information gleaned from running the instrumented binaries. The Mozilla build system will run both passes for you, you simply need to provide a script to run the application through some profiling scenarios in between the build passes. {{ Note("The script does not need to be automated! If you are not running your builds in an automated environment, you can simply have it launch the browser, then interact with the browser to generate the profiling data and close it when you\'re done.") }}</p>
<h2 id="Build_Configuration" name="Build_Configuration">Build Configuration</h2>
<p>You will need to setup a mozconfig as normal with the build options you want. You will also need to specify one additional option in your mozconfig for the PGO build process, the script to run between build passes. You set this with <code>mk_add_options PROFILE_GEN_SCRIPT</code> like so:</p>
<pre class="eval"> mk_add_options PROFILE_GEN_SCRIPT=/home/user/run-firefox.sh
</pre>
<p>This command will be run directly from make, so ensure that the file is executable. On Windows you will need to specify the interpreter name in front of the script, like so:</p>
<pre class="eval"> mk_add_options PROFILE_GEN_SCRIPT="sh /home/user/run-firefox.sh"
</pre>
<p>The script does not have to be a shell script, it can be Perl or Python or anything else that you can run from the command line, just make sure to specify the right command for running it.</p>
<p>You can use the same script the nightly builds use with the following:</p>
<pre class="eval"> mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'
</pre>
<h2 id="The_profile_generation_script" name="The_profile_generation_script">The profile generation script</h2>
<p>When your script is run it is passed in an environment variable, <code>OBJDIR</code>, containing the path to the object directory. A simple shell script for Linux that just launches the browser would look like this:</p>
<pre class="eval"> #!/bin/sh
 export NO_EM_RESTART=1
 mkdir $OBJDIR/_profileprofile
 $OBJDIR/dist/bin/firefox -no-remote -profile $OBJDIR/_profileprofile
</pre>
<p>It simply creates an empty profile directory, then runs Firefox with that profile directory. You would have to manually run your profiling scenarios and close the browser.</p>
<h2 id="Building" name="Building">Building</h2>
<p>PGO builds require you to start your build from a different target in client.mk. Instead of the usual <code>make -f client.mk build</code>, you simply do <code>make -f client.mk profiledbuild</code>. {{ Note("PGO builds will take much longer than normal builds! With GCC, you will wind up doing two complete clobber builds. With MSVC, you will only do one full build, and then another pass of re-linking, but re-linking libxul can take over a half hour on a fast machine, and use over one gigabyte of RAM.") }}{{ Warning("Linking a PGO build is known to exhaust the 32 bit address space on Win32.  To work around this either use a 64 bit compiler or use the /3GB boottime switch to give the linker more address space.  See <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=543034" title="https://bugzilla.mozilla.org/show_bug.cgi?id=543034">Bug 543034</a> for details.") }}</p>
Revert to this revision