What is Address Sanitizer?
Address Sanitizer (ASan) is a fast memory error detector that detects use-after-free and out-of-bound bugs in C/C++ programs. It uses a compile-time instrumentation to check all reads and writes during the execution. In addition, the runtime part replaces the
free functions to check dynamically allocated memory. More information on how ASan works can be found on the Address Sanitizer wiki.
The easiest way to get Firefox builds with Address Sanitizer is to use the public builds of mozilla-central, available for download:
- mozilla-central optimized builds (recommended for testing)
- mozilla-central debug builds (recommended for debugging if the optimized builds don't do the job)
These builds are updated and tested at least daily.
Creating your own Linux Try build
If for some reason you cannot use the pre-built packages mentioned in the last section (e.g. need to test a patch), you can either build Firefox yourself (see the section Manual build after this section) or use the try server to create the customized build for you. Pushing to try requires L1 commit access. If you don't have this access yet you can request access (see http://www.mozilla.org/hacking/committer/ and http://www.mozilla.org/hacking/commit-access-policy/ for the requirements). Note that this kind of access is mainly for developers and other regular contributors.
The tree contains several mozconfig files that can be readily used to create builds:
# Linux builds browser/config/mozconfigs/linux64/debug-asan - 64 bit debug build (debug+opt) browser/config/mozconfigs/linux64/nightly-asan - 64 bit release build # MacOSX build browser/config/mozconfigs/macosx64/debug-asan - 64 bit debug build (debug+opt)
All of these files refer to the common base configuration in
build/unix/mozconfig.asan which contains compiler information and flags. No changes should be required there anymore.
For Linux builds, these configuration files will be used by the
linux64-asan target. If you want to create a MacOSX build, you need to copy the configuration over the regular debug configuration before pushing to try:
cp browser/config/mozconfigs/macosx64/debug-asan browser/config/mozconfigs/macosx64/debug
If you haven't setup your
.hgrc for the try server yet, make sure you have these entries in
[extensions] mq = [paths] try = ssh://hg.mozilla.org/try
Now you go the regular way to get your try builds:
# Create new patch with our changes (or an empty patch in case of linux64) hg qnew try # Create debug and release build (-b do) for linux64 only hg qref -m 'try: -b do -e -p linux64-asan -u none -t none' # Push to try hg push -f try
If you want to create a MacOSX build instead, use 'try: -b d -e -p macosx64 -u none -t none' instead for the try message. You will receive an email for every build job started. After some time, you will receive a second email, containing the path to the uploaded build (the message subject might indicate an error but the build should have been uploaded anyway, except if there was some compile error. In that case, check the build log).
Warning: MacOSX builds are not working right now due to various bugs. See bug 1026162 (tracking bug) for current issues with these builds.
Warning: MacOSX builds currently suffer from a startup crash (see bug 935795). If you are encountering this crash, the only workaround is to add
to CFLAGS and CXXFLAGS in your mozconfig when compiling. This however disables all ASan stack checking which is a severe disadvantage. If possible, use Linux for testing instead.
The ASan instrumentation is implemented as an LLVM pass and integrated into Clang. Therefore, we must first get and build LLVM and Clang.
Note: The following instructions (for Linux) work with GCC 4.9 and G++ 4.9, please install those packages beforehand (e.g. gcc-4.9 and g++-4.9 via apt-get).
The following commands do a fresh SVN checkout/build of LLVM, Clang and the compiler runtime libraries with a revision confirmed to work:
# Latest revision confirmed to build with Linux # For MacOSX, please use at least r214699. REV=266460 svn co -r $REV https://llvm.org/svn/llvm-project/llvm/trunk llvm cd llvm export LLVM_HOME=`pwd` # We will refer to this variable later during the build. (cd tools && svn co -r $REV https://llvm.org/svn/llvm-project/cfe/trunk clang) (cd projects && svn co -r $REV https://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt)
It is NOT advised to use system-supplied versions of clang for ASAN builds; they may have issues or instabilities when used to build/test Firefox. The versions given are known to work.
Now we can start our build, which can take a while depending on your machine specs. In order to setup the build with the recommended
cmake method, you need to have at least cmake 2.8.8 installed. Adjust the
-j parameter to your needs depending on your CPUs (Linux users with a default gcc greater than 4.9 may have to call gcc-4.9 & g++-4.9 by prepending "CC=/usr/bin/gcc-4.9 CXX=/usr/bin/g++-4.9" to the cmake line; MacOSX users may need to add "
-DLLVM_ENABLE_LIBCXX=ON" to the cmake line):
mkdir build cd build cmake -DCMAKE_BUILD_TYPE:STRING=Release $LLVM_HOME make -j16
Building with cmake is highly recommended. In case you cannot build with cmake for some reason, it is still possible to build without it. Instead, use these steps then for building:
# Use these steps only if the cmake build steps cannot be used mkdir build (cd build && ../configure --enable-optimized && make -j 10 ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1)
Please note that when building without
cmake, the path to your resulting binaries will be different. For example, with cmake, the
clang binary will be at $LLVM_HOME/build/bin/clang while with the other build method, you will get it at $LLVM_HOME/build/Release/bin/clang.
Getting the source
The following build process was last tested with mozilla-central revision ad79ffdf94a3. Using that or any later revision, all you need to do is to get yourself a clone of mozilla-central.
Adjusting the build configuration
Create the build configuration file
.mozconfig with the following content in your mozilla-central directory:
# Combined .mozconfig file for ASan on Linux+Mac mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-asan # Adjust this to the number of CPU cores + 2 mk_add_options MOZ_MAKE_FLAGS=-j10 # Enable ASan specific code and build workarounds ac_add_options --enable-address-sanitizer # Ensure you set this to your LLVM_HOME path export LLVM_HOME="/path/to/your/llvm" # Set CC/CXX based on LLVM_HOME export CC="$LLVM_HOME/build/bin/clang" export CXX="$LLVM_HOME/build/bin/clang++" # This will ensure the symbolizer is packaged with the binary export LLVM_SYMBOLIZER="$LLVM_HOME/build/bin/llvm-symbolizer" # Add ASan to our compiler flags export CFLAGS="-fsanitize=address -Dxmalloc=myxmalloc -fPIC" export CXXFLAGS="-fsanitize=address -Dxmalloc=myxmalloc -fPIC" # Additionally, we need the ASan flag during linking. Normally, our C/CXXFLAGS would # be used during linking as well but there is at least one place in our build where # our CFLAGS are not added during linking. # Note: The use of this flag causes Clang to automatically link the ASan runtime :) export LDFLAGS="-fsanitize=address" # These three are required by ASan ac_add_options --disable-jemalloc ac_add_options --disable-crashreporter ac_add_options --disable-elf-hack # Keep symbols to symbolize ASan traces later export MOZ_DEBUG_SYMBOLS=1 ac_add_options --enable-debug-symbols ac_add_options --disable-install-strip # Settings for an opt build (preferred) # The -gline-tables-only ensures that all the necessary debug information for ASan # is present, but the rest is stripped so the resulting binaries are smaller. ac_add_options --enable-optimize="-O2 -gline-tables-only" ac_add_options --disable-debug # Settings for a debug+opt build #ac_add_options --enable-optimize #ac_add_options --enable-debug # MacOSX only: Uncomment and adjust this path to match your SDK # ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
You may also need this, as seen in
browser/config/mozconfigs/linux64/nightly-asan (the config file used for Address Sanitizer builds used for automated testing):
# ASan specific options on Linux ac_add_options --enable-valgrind
Starting the build process
Now you start the build process using the regular
./mach build command.
Note: if the build fails with an error about 'max_align_t', comment out the line in <cstddef> that uses it (e.g. cstddef:51 "using ::max_align_t;" in Fedora 21 with libstdc++ 4.9.2 Release : 6.fc21). clang 200213 doesn't have max_align_t support.
Warning: If your MacOSX builds fails with fatal error: 'cstdio' file not found (or 'algorithm', 'string', and something from standard library), you need to install libc++ by yourself.
# Install libc++ cd llvm/projects svn co -r $REV http://llvm.org/svn/llvm-project/libcxx/trunk libcxx svn co -r $REV http://llvm.org/svn/llvm-project/libcxxabi/trunk libcxxabi # Re-build LLVM cd .. && rm build && mkdir build && cd build cmake -DCMAKE_BUILD_TYPE:STRING=Release $LLVM_HOME make -j16
After the build has completed, you can start Firefox from the
objdir as usual.
XRE_NO_WINDOWS_CRASH_DIALOG=1before starting, this doesn't seem to be required anymore.
LLVM_ROOT to match your setup. Once you have adjusted everything, execute this script in the
js/src/ subdirectory and pass a directory name as the first parameter. The build will then be created in a new subdirectory with that name.
#! /bin/sh if [ -z $1 ] ; then echo "usage: $0 <dirname>" elif [ -d $1 ] ; then echo "directory $1 already exists" else autoconf2.13 mkdir $1 cd $1 LLVM_ROOT="/path/to/llvm" CC="$LLVM_ROOT/build/Release/bin/clang" \ CXX="$LLVM_ROOT/build/Release/bin/clang++" \ CFLAGS="-fsanitize=address" \ CXXFLAGS="-fsanitize=address" \ LDFLAGS="-fsanitize=address" \ ../configure --enable-debug --enable-optimize --enable-address-sanitizer --disable-jemalloc make -j 8 fi
Getting Symbols in Address Sanitizer Traces
By default, ASan traces are unsymbolized and only print the binary/library and a memory offset instead. In order to get more useful traces, containing symbols, there are two approaches.
Using the LLVM Symbolizer (recommended)
LLVM ships with a symbolizer binary that ASan will readily use to immediately output symbolized traces. To use it, just set the environment variable
ASAN_SYMBOLIZER_PATH to reflect the location of your
llvm-symbolizer binary, before running the process. If you downloaded a prebuilt package or made a try build, then the binary will be included in the tarball (same location as the
firefox binary). If you've built ASan yourself, then you will want to set the variable to
Post-Processing Traces with asan_symbolize.py
Instead of using the llvm-symbolizer binary, you can also pipe the output through the
asan_symbolize.py script, shipped with LLVM (
$LLVM_HOME/projects/compiler-rt/lib/asan/scripts/asan_symbolize.py). The disadvantage is that the script will need to use
addr2line to get the symbols, which means that every library will have to be loaded into memory (including
libxul, which takes a bit).
However, in certain situations it makes sense to use this script. For example, if you have/received an unsymbolized trace, then you can still use the script to turn it into a symbolized trace, given that you can get the original binaries that produced the unsymbolized trace. In order for the script to work in such cases, you need to ensure that the paths in the trace point to the actual binaries, or change the paths accordingly.
Since the output of the
asan_symbolize.py script is still mangled, you might want to pipe the output also through
Troubleshooting / Known problems
Cannot specify -o when generating multiple output files
If you get the error "
cannot specify -o when generating multiple output files" from clang, disable
elf-hack in your
mozconfig to work around the issue:
Since an issue with -O2/-Os and ASan has been resolved, the regular optimizations used by Firefox should work without any problems. The optimized build has only a barely noticable speed penalty and seems to be even faster than regular debug builds.
Crashes on Ubuntu 11.10
There seems to be a bug on Ubuntu 11.10 where binaries such as
shlibsign and other NSS tools just crash on startup. In conjunction you'll see warnings like
/usr/bin/ld.bfd.real: /path/to/llvm/build/Release+Asserts/bin/../lib/clang/3.1/lib/linux/libclang_rt.asan-x86_64.a(asan_thread_registry.o)(.text+0x2d6): unresolvable R_X86_64_PLT32 relocation against symbol `memcpy@@GLIBC_2.2.5'
during the build. The exact cause is currently not known but it might be a bug in
ld on that Ubuntu version, as using
gold as the linker resolves the problem. To work around this problem, just do
sudo apt-get install binutils-gold
gold will be your default linker. This comes with the positive side effect that your build will be a few minutes faster due to less linking time required.
Recent cmake on Ubuntu 12.04 LTS
Unfortunately, the current Ubuntu LTS comes with a
cmake version that is too old for building compiler-rt (2.8.7). There are several PPAs available though that have newer
cmake available, e.g. https://launchpad.net/~irie/+archive/cmake . This PPA has been confirmed to work, but use at your own risk of course.
No "AddressSanitizer: libc interceptors initialized" shows after running ./mach run
$ ASAN_OPTIONS=verbosity=2 ./mach run
Use the above command instead
"An admin user name and password" is required to enter Developer Mode
Please enable Developper mode by:
$ /usr/sbin/DevToolsSecurity -enable Developer mode is now enabled.
Debugging issues that ASan finds
When ASan discovers an issue it will simply print an error message and exit the app. To stop the app in a debugger before ASan exits it, set a breakpoint on ReportGenericError (used to be __asan_report_error). For more info on using ASan and debugging issues that it uncovers, see the Google Code Address Sanitizer wiki page.
LeakSanitizer (LSan) is a special execution mode for regular ASan. It takes advantage of how ASan tracks the set of live blocks at any given point to print out the allocation stack of any block that is still alive at shutdown, but is not reachable from the stack, according to a conservative scan. This is very useful for detecting leaks of things such as
char* that do not participate in the usual Gecko shutdown leak detection.
To make an ASan build run LSan, set the environment variable
detect_leaks=1 (or add it as an entry to a
:-separated list if it is already set to something). If you are using a non-debug build, you will also want to set the environment variable
MOZ_CC_RUN_DURING_SHUTDOWN=1, to ensure that we run shutdown GCs and CCs to avoid spurious leaks.
If an object that is reported by LSan is intentionally never freed, a symbol can be added to
build/sanitizers/lsan_suppressions.txt to get LSan to ignore it.
For some more information on LSan, see the Google Code Leak Sanitizer wiki page.