This article is in need of a technical review.
What is Thread Sanitizer?
Thread Sanitizer (TSan) is a fast data race detector for C/C++ programs. It uses a compile-time instrumentation to check all non-race-free memory access at runtime. Unlike other tools, it understands compiler-builtin atomics and synchronization and therefore provides very accurate results with no real false positives. More information on how TSan works can be found on the Thread Sanitizer wiki.
Note: No public builds are available at this time yet.
The TSan instrumentation is implemented as an LLVM pass and integrated into Clang. The version of Clang you need to use depends on what Firefox release you are compiling
- If you are building from mozilla-central, you must use Clang 3.6 or later. Clang 3.4 + TSan cannot compile Firefox due to an internal compiler error; some versions of Clang 3.5 packaged by Linux distros do not work for the same reason. You can check which version of clang you have by running the command:
- If you are not building from mozilla-central and are building Firefox 39 or earlier, you must use Clang 3.3. You can find precompiled binaries for LLVM/Clang 3.3 on the LLVM releases page.
We recommend using Clang 3.6 or later.
Getting the source
If you don't have a source code repository clone yet, you need to get yourself a clone of mozilla-central.
Adjusting the build configuration
Create the build configuration file
.mozconfig with the following content in your mozilla-central directory:
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-tsan mk_add_options MOZ_MAKE_FLAGS=-j12 # Enable LLVM specific code and build workarounds ac_add_options --enable-thread-sanitizer # If clang is already in your $PATH, then these can simply be: # export CC=clang # export CXX=clang++ export CC="/path/to/clang" export CXX="/path/to/clang++" # llvm-symbolizer displays much more complete backtraces when data races are detected. # If it's not already in your $PATH, then uncomment this next line: #export LLVM_SYMBOLIZER="/path/to/llvm-symbolizer" # Add TSan to our compiler flags export CFLAGS="-fsanitize=thread -fPIC -pie" export CXXFLAGS="-fsanitize=thread -fPIC -pie" # Additionally, we need the TSan flag during linking. Normally, our C/CXXFLAGS would # be used during linking as well but there is at least one place in our build where # our CFLAGS are not added during linking. # Note: The use of this flag causes Clang to automatically link the TSan runtime :) export LDFLAGS="-fsanitize=thread -fPIC -pie" # These three are required by TSan ac_add_options --disable-jemalloc ac_add_options --disable-crashreporter ac_add_options --disable-elf-hack # Keep symbols to symbolize TSan traces export MOZ_DEBUG_SYMBOLS=1 ac_add_options --enable-debug-symbols ac_add_options --disable-install-strip # Settings for an opt build ac_add_options --enable-optimize="-O2 -gline-tables-only" ac_add_options --disable-debug
Starting the build process
Now you start the build process using the regular
make -f client.mk command.
After the build has completed, you can start Firefox from the
objdir as usual.
LLVM_ROOT to match your setup. Once you have adjusted everything, execute this script in the
js/src/ subdirectory and pass a directory name as the first parameter. The build will then be created in a new subdirectory with that name.
#! /bin/sh if [ -z $1 ] ; then echo "usage: $0 <dirname>" elif [ -d $1 ] ; then echo "directory $1 already exists" else autoconf2.13 mkdir $1 cd $1 LLVM_ROOT="/path/to/llvm" CC="$LLVM_ROOT/build/bin/clang" \ CXX="$LLVM_ROOT/build/bin/clang++" \ CFLAGS="-fsanitize=thread -fPIC -pie" \ CXXFLAGS="-fsanitize=thread -fPIC -pie" \ LDFLAGS=""-fsanitize=thread -fPIC -pie" \ ../configure --disable-debug --enable-optimize="-O2 -gline-tables-only" --enable-llvm-hacks make -j 8 fi
Using LLVM Symbolizer for faster/better traces
By default, TSan traces are symbolized because otherwise, the runtime suppression list wouldn't work. The default way of symbolizing the traces is much slower than using
llvm-symbolizer for that purpose:
LLVM ships with a symbolizer binary that TSan will readily use to immediately output symbolized traces much faster. Furthermore, additional debug information is included in traces when
llvm-symbolizer is used. To use it, either make sure the
llvm-symbolizer binary is in your PATH or set the environment variable
TSAN_OPTIONS="external_symbolizer_path=/path/to/llvm-symbolizer" before running the process. If you've built TSan as described above, then you will want to set the variable to
Ignoring known races
In some cases, races can be confirmed to be benign and fixing them would cause possible performance issues. TSan offers two possibilities to ignore such known races.
Functions with racy memory access can be flagged, such that the compiler will not instrument them. In our codebase, this is done using the MOZ_TSAN_BLACKLIST macro on the function definition. It will expand to the appropriate attributes if the compiler supports them. Compile-time blacklisting is the preferred way of blacklisting, because it does not have a performance impact. Still, blacklisting should be done very carefully so we won't miss important bugs.
To prevent races from showing up at runtime, TSan also provides a runtime suppression list. This list is a good tool to temporarily hide some traces, especially for debugging purposes. However, unlike compile-time blacklisting, runtime suppressions do have a performance impact, especially if the race in question keeps showing up a lot. For more information on how to use the runtime suppression list, see the TSan wiki.