Debugging Mozilla with Valgrind

This page describes how to use Valgrind's Memcheck tool to find memory errors.  For details on using Valgrind's Helgrind tool to find data races and other threading errors, see here.

Note: Valgrind + Firefox does not work on Mac OS 10.8.  See bug 834078.

Where can I get Valgrind?

Linux: Download Valgrind directly, or use your distribution's package manager.

Mac: Get Valgrind trunk from SVN and build it.

Make sure you have version 3.7.0 or later of Valgrind.  Newer versions tend to have better compatibility with both Firefox's JITs and newer toolchain components (compiler, libc and linker versions).

Where can I get support?

If Valgrind asserts, crashes, doesn't do what you expect, or otherwise acts up, first of all try using the Valgrind trunk from SVN.  Oftentimes bugs are fixed in the trunk before most users fall across them.  If that doesn't help, consider filing a bug report, and/or mailing Julian Seward or Nick Nethercote.

What's the bare minimum I need to know to get started?

Five things:

(1) Build Firefox with --disable-jemalloc and --enable-valgrind, by adding "ac_add_options --disable-jemalloc" and "ac_add_options --enable-valgrind" to your mozconfig.  You need both flags.  If you don't do this, the results you'll get from Valgrind will be total nonsense.

(2) Run Valgrind with --smc-check=all-non-file and --vex-iropt-register-updates=allregs-at-mem-access. Always. If you don't do this, it will quickly crash in JIT-generated code.

(3) On MacOS, also run with --dsymutil=yes.  If you don't do this, you won't get line number information in error messages.

(4) Read the quick-start guide.  It's very short.

(5) Try to find time to read the rest of this page.  You may find it helpful.

Toolchain caveats (updated 25 Jan 2013)

Valgrind crahes when running Firefox on Mac OS 10.8 (Mountain Lion).  Linux and older versions of Mac OS are unaffected.  See bug 834078.

Code linked by GNU gold may not work with Valgrind 3.6.x; Valgrind may error out with "Serious error reading debug info, can't make sense of .eh_frame section mapping".  This issue should be fixed in Valgrind trunk, rev 11790 (30 May 2011).

Code compiled by LLVM (that is, XCode 4, but not XCode 3) at high optimisation levels causes Memcheck to report false uninitialised value errors, but otherwise works OK.  This is a known issue that will be fixed in Valgrind 3.7.0.  See here for an easy workaround.

Valgrind built using XCode 4 may segfault at startup.  This was fixed in trunk svn rev 11686 (6 Apr 2011).  See here for background.

Is there a shared Memcheck suppression file for known bugs?

/build/valgrind/ contains the suppression files used by the periodic Valgrind jobs on Tinderbox. Some of these files are platform-specific.

Jesse has a larger suppression that he uses for fuzzing, but it is out of date.

What do I do if the JIT crashes on startup?

Pass the parameter --smc-check=all-non-file to valgrind.

Note: You should use --smc-check=all-non-file when using all Valgrind tools -- Memcheck, Helgrind, Massif, DHAT, whatever.  Firefox will crash without it, if it loads a page and the JS JIT is enabled.  Configuring Firefox with --enable-valgrind is not enough to make that work. --smc-check=all-non-file is always required.

How Do I Run A Mochitest Under Valgrind?

make mochitest-plain TEST_PATH=relative/path/test_mything.html EXTRA_TEST_ARGS='--debugger=valgrind --setpref=javascript.options.jit.chrome=false --setpref=javascript.options.jit.content=false'

See the Mochitest docs for more information about running mochitests.

You may want to add --debugger-args=--trace-children=yes to EXTRA_TEST_ARGS if you want Valgrind to trace into the OOPP helper process(es).

As per comments above you must also pass --debugger-args=--smc-check=all-non-file in EXTRA_TEST_ARGS.

As of March 2011, using the tips in the next section, it is possible to do a complete run of mochitests-plain (all 200k tests) on Valgrind in about 15 CPU hours on a Core i5 machine.  Maximum process size is 5.4G, of which about 80% is in memory.  Runs of small subsets of mochitests take far less memory.  This level of resource usage puts it within easy reach of automated nightly testing on individual developer machines.

Tips for improving performance and accuracy of Valgrind's Memcheck tool

Running Firefox on Valgrind's Memcheck tool can be a frustratingly slow experience.  But there are things you can do to improve this.  None of the following is by itself a silver bullet, but taken together they do help considerably.  A sample mozconfig file incorporating all the suggestions is shown below.

(1) MANDATORY: build Firefox with JEMalloc disabled ("ac_add_options --disable-jemalloc").  Despite considerable efforts, Memcheck does not adequately understand JEMalloc's behaviour, and you will be swamped by false error reports if you use it.  Also, you will lose most of Memcheck's error detection capabilities.  Hence it is pretty much mandatory to use the standard system implementation of malloc/free/new/delete if you want sensible results.

(2) MANDATORY: build Firefox with Valgrind hints enabled ("ac_add_options --enable-valgrind").  If you don't do this, Memcheck will go nuts when it sees the JS engine's garbage collector scanning the thread stacks.

(3) MANDATORY: run Valgrind with --smc-check=all-non-file, always.  Otherwise it will crash in JIT-created code.  Also, some of the MacOSX system libraries generate code on the fly, and will not run correctly without this flag.  On ARM CPUs you can omit it, since ARM requires explicit icache invalidation and Valgrind observes and honours such requests.

(4) MANDATORY for MacOS: run Valgrind with --dsymutil=yes, always.  Otherwise you won't get line number information in error messages.  See also (10) below.

(5) MANDATORY for Linux: run with environment variable "G_SLICE=always-malloc".  This is necessary to get the Gnome system libraries (I think) to use plain malloc, instead of pool allocators.

The rest of the points are not mandatory, but you'd nevertheless be wise to pay attention to them.

(6) Use a decent machine.  Valgrind is tremendously memory-intensive, so the single most important factor is having a large level 2 cache.  2MB is a bare minimum, 4MB or more is preferable.  Most mid-to-upper range Intel Core 2s and Core iXs have 4MB or larger L2/L3s and work well.  I'd guess the latest mid-to-upper-range AMDs are also good, although I haven't tried them recently.  RAM is also important: 2GB of RAM is a bare minimum, and 4GB is better.

(7)  Build Firefox with "-g -O".  Don't use a plain "-g" (unoptimized) build.  Checking memory references takes Valgrind a lot of time.  At -O0 (no optimization), gcc does't do much register allocation, so the generated code has many unnecessary memory references which slow Valgrind down.  At -O (that is, -O1) most of those disappear, whilst retaining pretty good stack-unwind-ability, so that Valgrind can still produce sane stack traces.  The difficulties with running optimised code on Memcheck are a somewhat increased risk of false positive uninitialised-value errors, and incomplete or incomprehensible stack traces.  At -O1 neither of these seem significant.  If you want to live dangerously, and you have gcc-4.3 or later, try "-g -O2".

(8) Use 64-bit builds of Firefox in preference to 32-bit builds.  64-bit code has more available registers and better calling conventions, both of which reduce the number of memory references Valgrind has to check.  Also, for SVN trunk Valgrind builds only, the overhead for --smc-check=all-non-file is lower than on 32-bit targets.

(9) Don't use --track-origins=yes unless you are hunting down a specific uninitialised-value error.  It pretty much halves the speed of Valgrind.  That said, it is still way faster than tracking down sources of uninitialised data by hand, and so constitutes a net programmer productivity win.

(10) Use Linux rather than MacOS.  Unfortunately, Valgrind has difficulties in with threaded code on MacOS, which sometimes cause it to run far slower than on Linux.  Fixing this in Valgrind will not be simple.  Such difficulties do not occur on Linux.  If your code is single threaded you can of course ignore this point.  Bear in mind that pretty much any startup of the browser plus a bit of surfing causes easily a couple of dozen threads to get created, so pretty much all browser startups will encounter this problem on MacOS.

(11) Use the latest Valgrind trunk from SVN.  It's easy to download and build.  The trunk sometimes contains optimizations and bug fixes not yet present in formal releases.  At a bare mininum, use the stock 3.6.1 rather than ancient versions (3.2.x, 3.3.x, etc).

Using all these together, on a Core i5 670 (3.46 GHz) running 64-bit Linux, I can surf the web, reading news sites over my morning coffee, whilst running on Memcheck.  The delays are such that it is obvious that Fx is not running natively, but they are small enough that I spend most of my time reading and not much time waiting for Fx.  Here's a recommended mozconfig:

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/ff-opt
ac_add_options --enable-tests
ac_add_options --enable-optimize="-g -O -freorder-blocks"
ac_add_options --disable-jemalloc
ac_add_options --enable-valgrind
mk_add_options MOZ_MAKE_FLAGS="-j4"

As per comments above, I've been experimenting recently with -O2 rather than "-O -freorder-blocks", for maximum effect.

Per-platform comments, current as of 23 March 2011

Linux on X86/AMD64/PPC32/PPC64/ARMv7

These work out of the box, either via the 3.6.1 release or from trunk sources.  The X86, AMD64 and ARMv7 ports are widely used.  We also support PPC32 and PPC64 on Linux.  This can be interesting in that Valgrinding on those platforms throws up the occasional endianness bug which otherwise might have gone unnoticed.

On ARMv7, PPC32 and PPC64, explicit icache invalidation for JIT generated code is mandatory.  Valgrind observes and honours the such directives, so you get transparent support for JIT-generated code without having to use --smc-check=all.  No such luck on X86 or AMD64 unfortunately.

On ARMv7 -- unlike on other platforms -- stack traces will end in any code not compiled with -g, so you need to compile everything with -g.  This doesn't mean you can't use optimisation, though -- so the standard recommendation of "-g -O" stands.

For un-released Linux distros (Fedora Rawhide, etc) you'll need the trunk sources, since fixes for the latest gcc and glibc versions appears there first.  Without them you'll be flooded with false errors from Memcheck, and have debuginfo reading problems.

Mac OS X 10.5.x and 10.6.x on X86 and AMD64

Valgrind 3.6.x works out of the box on these platforms.  You need run permanently with --dsymutil=yes, otherwise you'll never get any line number info.  See caveat (10) above re performance on heavily threaded code.

Windows on X86 (32-bit)

Valgrind doesn't support Windows directly.  However, if you want to live right on the leading edge, it is possible to run a debug Win32 build of Fx on Wine (trunk) on Valgrind (trunk), and get sensible results.  Valgrind contains a rudimentary PDB file reader, so you can get source locations in MSVC compiled code.  A couple of bugs in Fx have been filed as a result of preliminary investigations using this setup.  The basic recipe is documented here, but it's all a bit flaky and needs further work.  If you're interested in this please contact Julian.  It's not as scary as it sounds, and I would like to get this to the status of being usable and useful.

11 Feb 2010: enough stuff works that I can now edit Fx sources on the Windows box, do "make -f client.mk build" on it, and then simply run the Fx/Wine/Valgrind stack on a Linux box.  Assuming of course that the Windows filesystem is visible on the Linux box, which is easy by running "mount -t cifs" on the Linux box.

06 Apr 2010: further improvements to the Wine/Valgrind parts of this stack have accumulated.  A handful of bugs have been found and fixed in Win32 specific Fx code as a result.  The next step is to run all of mochitests-plain on this stack, as has been done for Linux in bug 549224.

Document Tags and Contributors

Last updated by: Sheppy,