What is Address Sanitizer?
Address Sanitizer (ASan) is a fast memory error detector that detects use-after-free and out-of-bound bugs in C/C++ programs. It uses a compile-time instrumentation to check all reads and writes during the execution. In addition, the runtime part replaces the
free functions to check dynamically allocated memory. More information on how ASan works can be found at on the Address Sanitizer wiki.
Try Server Builds
The easiest way to get Firefox builds with Address Sanitizer is to use the regular try builds, provided here for download. These builds are updated daily and help to save important build resources on the try server.
Creating your own Linux Try build
If for some reason you cannot use the pre-built packages mentioned in the last section (e.g. need to test a patch), you can either build Firefox yourself (see the section Manual build after this section) or use the try server to create the customized build for you. Pushing to try requires L1 commit access. If you don't have this access yet you can request access (see http://www.mozilla.org/hacking/committer/ and http://www.mozilla.org/hacking/commit-access-policy/ for the requirements). Note that this kind of access is mainly for developers and other regular contributors.
The tree contains several mozconfig files that can be readily used to create builds:
# Linux builds browser/config/mozconfigs/linux64/debug-asan - 64 bit debug build (debug+opt) browser/config/mozconfigs/linux64/nightly-asan - 64 bit release build browser/config/mozconfigs/linux32/debug-asan - 32 bit debug build (debug+opt) browser/config/mozconfigs/linux32/nightly-asan - 32 bit release build # MacOSX build browser/config/mozconfigs/macosx64/debug-asan - 64 bit debug build (debug+opt)
Please note that using 32 bit linux builds is not recommended and might lead to problems due to the limited amount of memory and stack space available. For MacOSX there are currently no universal builds or 32 bit builds available because of unresolved problems with the build system. All of these files refer to the common base configuration in
build/unix/mozconfig.asan which contains compiler information and flags. No changes should be required there anymore.
However, in order for the necessary toolchain (Clang) to be available, you need to copy the required manifest file(s):
cp browser/config/tooltool-manifests/linux32/clang.manifest browser/config/tooltool-manifests/linux32/releng.manifest cp browser/config/tooltool-manifests/linux64/clang.manifest browser/config/tooltool-manifests/linux64/releng.manifest cp browser/config/tooltool-manifests/macosx64/clang.manifest browser/config/tooltool-manifests/macosx64/releng.manifest
Once you have done that, copy the desired ASan mozconfig files over the regular mozconfigs used during try builds:
cp browser/config/mozconfigs/linux64/debug-asan browser/config/mozconfigs/linux64/debug cp browser/config/mozconfigs/linux64/nightly-asan browser/config/mozconfigs/linux64/nightly cp browser/config/mozconfigs/linux32/debug-asan browser/config/mozconfigs/linux32/debug cp browser/config/mozconfigs/linux32/nightly-asan browser/config/mozconfigs/linux32/nightly cp browser/config/mozconfigs/macosx64/debug-asan browser/config/mozconfigs/macosx64/debug
If you haven't setup your
.hgrc for the try server yet, make sure you have these entries in
[extensions] mq = [paths] try = ssh://hg.mozilla.org/try
Now you go the regular way to get your try builds:
# Create new patch with our changes hg qnew asan-try.patch browser/config/mozconfigs/ build/unix/mozconfig.asan # Create debug and release build (-b do) for linux64 only hg qref -m 'try: -b do -e -p linux64 -u none -t none' # Push to try hg push -f try
If you want to create a MacOSX build instead, use 'try: -b d -e -p macosx64 -u none -t none' instead for the try message. You will receive an email for every build job started. After some time, you will receive a second email, containing the path to the uploaded build (the message subject might indicate an error but the build should have been uploaded anyway, except if there was some compile error. In that case, check the build log).
The ASan instrumentation is implemented as an LLVM pass and integrated into Clang. Therefore, we must first get and build LLVM and Clang.
The following commands do a fresh SVN checkout/build of LLVM, Clang and the compiler runtime libraries with a revision confirmed to work:
# Latest revision confirmed to build with Linux and MacOSX (10.7 and 10.8). REV=163716 svn co -r $REV http://llvm.org/svn/llvm-project/llvm/trunk llvm cd llvm export LLVM_HOME=`pwd` # We will refer to this variable later during the build. (cd tools && svn co -r $REV http://llvm.org/svn/llvm-project/cfe/trunk clang) (cd projects && svn co -r $REV http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt)
Now we can start our build, which can take a while depending on your machine specs. Adjust the
-j parameter to your needs depending on your CPUs:
mkdir build (cd build && ../configure --enable-optimized && make -j 10)
ASan runtime library
Parts of the ASan code are in a runtime library which is included now in the compiler-rt repository that we checked out and built earlier. The default installation target builds a
.a file that is linked to all executables built with Clang (but not to shared objects, which is why they will have unresolved symbols). To build the runtime library we perform the following steps:
cd $LLVM_HOME/projects/compiler-rt/lib/asan make -f Makefile.old get_third_party make -f Makefile.old test -j 10 make -f Makefile.old install
Getting the source
The following build process was last tested with mozilla-central revision ad79ffdf94a3. Using that or any later revision, all you need to do is to get yourself a clone of mozilla-central.
Adjusting the build configuration
Create the build configuration file
.mozconfig with the following content in your mozilla-central directory:
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-asan mk_add_options MOZ_MAKE_FLAGS=-j12 # Enable ASan specific code and build workarounds ac_add_options --enable-address-sanitizer # Ensure you set this to your LLVM_HOME path export LLVM_HOME="/path/to/your/llvm" # Set CC/CXX based on LLVM_HOME export CC="$LLVM_HOME/build/Release+Asserts/bin/clang" export CXX="$LLVM_HOME/build/Release+Asserts/bin/clang++" # Add ASan to our compiler flags, including blacklist export CFLAGS="-faddress-sanitizer -Dxmalloc=myxmalloc" export CXXFLAGS="-faddress-sanitizer -Dxmalloc=myxmalloc" # Additionally, we need the ASan flag during linking. Normally, our C/CXXFLAGS would # be used during linking as well but there is at least one place in our build where # our CFLAGS are not added during linking. # Note: The use of this flag causes Clang to automatically link the ASan runtime :) export LDFLAGS="-faddress-sanitizer" # Avoid using ASan flags when building host tools like nsinstall export HOST_CFLAGS=" " export HOST_CXXFLAGS=" " export HOST_LDFLAGS=" " # These two are required by ASAn ac_add_options --disable-jemalloc ac_add_options --disable-crashreporter # Keep symbols to symbolize ASan traces later export MOZ_DEBUG_SYMBOLS=1 ac_add_options --enable-debug-symbols ac_add_options --disable-install-strip # Settings for a debug+opt build ac_add_options --enable-optimize ac_add_options --enable-debug # Settings for an opt build #ac_add_options --enable-optimize #ac_add_options --disable-debug
Starting the build process
Now you start the build process using the regular
make -f client.mk command.
After the build has completed, you can start Firefox from the
objdir as usual.
XRE_NO_WINDOWS_CRASH_DIALOG=1before starting, this doesn't seem to be required anymore.
LLVM_ROOT to match your setup. Once you have adjusted everything, execute this script in the
js/src/ subdirectory and pass a directory name as the first parameter. The build will then be created in a new subdirectory with that name.
#! /bin/sh if [ -z $1 ] ; then echo "usage: $0 <dirname>" elif [ -d $1 ] ; then echo "directory $1 already exists" else autoconf2.13 mkdir $1 cd $1 LLVM_ROOT="/path/to/llvm" CC="$LLVM_ROOT/build/Release+Asserts/bin/clang" \ CXX="$LLVM_ROOT/build/Release+Asserts/bin/clang++" \ CFLAGS="-faddress-sanitizer -Dxmalloc=myxmalloc" \ CXXFLAGS="-faddress-sanitizer -Dxmalloc=myxmalloc" \ ../configure --enable-debug --disable-optimize --enable-address-sanitizer make -j 8 fi
Troubleshooting / Known problems
No symbols in crash traces
As the ASan homepage already explains, ASan does not (yet) have an in-process symbol resolver. Therefore, all crash traces will only contain addresses. fortunately, ASan comes with a python script, that will resolve the addresses to files, lines and mangled function names. You can find it at
You can then run the result through
c++filt to get readable function names.
Cannot specify -o when generating multiple output files
If you get the error "
cannot specify -o when generating multiple output files" from clang, disable
elf-hack in your
mozconfig to work around the issue:
Since an issue with -O2/-Os and ASan has been resolved, the regular optimizations used by Firefox should work without any problems. The optimized build has only a barely noticable speed penalty and seems to be even faster than regular debug builds.
Crashes on Ubuntu 11.10
There seems to be a bug on Ubuntu 11.10 where binaries such as
shlibsign and other NSS tools just crash on startup. In conjunction you'll see warnings like
/usr/bin/ld.bfd.real: /path/to/llvm/build/Release+Asserts/bin/../lib/clang/3.1/lib/linux/libclang_rt.asan-x86_64.a(asan_thread_registry.o)(.text+0x2d6): unresolvable R_X86_64_PLT32 relocation against symbol `memcpy@@GLIBC_2.2.5'
during the build. The exact cause is currently not known but it might be a bug in
ld on that Ubuntu version, as using
gold as the linker resolves the problem. To work around this problem, just do
sudo apt-get install binutils-gold
gold will be your default linker. This comes with the positive side effect that your build will be a few minutes faster due to less linking time required.
Debugging issues that ASan finds
When ASan discovers an issue it will simply print an error message and exit the app. To stop the app in a debugger before ASan exits it, set a breakpoint in __asan_report_error. For more info on using ASan and debugging issues that it uncovers, see the Google Code Address Sanitizer wiki page.