This page describes how to use Valgrind's Memcheck tool to find memory errors.  For details on using Valgrind's Helgrind tool to find data races and other threading errors, see here.

Supported targets (updated 13 Jan 2015)

Linux on desktop (x86_64, x86) works well.  More generally, Linux on practically anything, including Android and Firefox OS, should work, although you may have to do some hoop jumping to come up with a usable combination.

MacOS X 10.10 (Yosemite), 64 bit only, works, although it can be a bit of a rough ride.  See "Per-platform comments" section below.

Where can I get Valgrind?

Linux: Download Valgrind directly, or use your distribution's package manager.

MacOSX: Get Valgrind trunk from SVN and build it.  Don't use 3.10.x or any other tarball.

Make sure you have version 3.10.1 or later of Valgrind.  Newer versions tend to have better compatibility with both Firefox's JITs and newer toolchain components (compiler, libc and linker versions).

Where can I get support?

If Valgrind asserts, crashes, doesn't do what you expect, or otherwise acts up, first of all read this page and make sure you have both Firefox and Valgrind correctly configured.  If that's all OK, try using the Valgrind trunk from SVN.  Oftentimes bugs are fixed in the trunk before most users fall across them.  If that doesn't help, consider filing a bug report, and/or mailing Julian Seward or Nick Nethercote.

What's the bare minimum I need to know to get started? (updated 22 Dec 2014)

Seven things:

(1) Build Firefox with --disable-jemalloc and --enable-valgrind, by adding "ac_add_options --disable-jemalloc" and "ac_add_options --enable-valgrind" to your mozconfig.  You need both flags.  If you don't do this, the results you'll get from Valgrind will be total nonsense.

(2) Run Valgrind with --smc-check=all-non-file and --vex-iropt-register-updates=allregs-at-mem-access. Always. If you don't do this, it will quickly crash in JIT-generated code.

(3) On MacOS, also run with --dsymutil=yes.  If you don't do this, you won't get line number information in error messages.

(4) On Linux, run with --read-inline-info=yes, as it makes it much easier to understand stack traces in the presence of inlining.

(5) Run Valgrind with --show-mismatched-frees=no, for reasons detailed below.  At least for now.

(6) Read the quick-start guide.  It's very short.

(7) Try to find time to read the rest of this page.  You may find it helpful.  For serious use of Valgrind, reading the rest of the page is pretty much mandatory.

Toolchain caveats (updated 13 Jan 2015)

Code compiled by LLVM (that is, XCode 4, but not XCode 3) at high optimisation levels causes Memcheck to report false uninitialised value errors, but otherwise works OK.  See here for an easy workaround.  Note that this workaround is enabled by default for MacOS builds of Valgrind, so this comment applies only if you are running LLVM compiled code on Linux.

Is there a shared Memcheck suppression file for known bugs?

/build/valgrind/ contains the suppression files used by the periodic Valgrind jobs on Tinderbox. Some of these files are platform-specific.

Jesse has a larger suppression that he uses for fuzzing, but it is out of date.

What do I do if the JIT crashes on startup?

Pass the flags --smc-check=all-non-file and --vex-iropt-register-updates=allregs-at-mem-access to valgrind.

Note: You should use --smc-check=all-non-file and --vex-iropt-register-updates=allregs-at-mem-access when using any Valgrind tool -- Memcheck, Helgrind, Massif, DHAT, whatever.  Firefox will crash without it, if it loads a page and the JS JIT is enabled.  Configuring Firefox with --enable-valgrind is not enough to make that work.  Both of these flags are also necessary.

How Do I Run A Mochitest Under Valgrind?

./mach mochitest-plain --debugger="valgrind" --debugger-args="--smc-check=all-non-file --vex-iropt-register-updates=allregs-at-mem-access" relative/path/to/tests

See the Mochitest docs for more information about running mochitests.

You may want to add --debugger-args=--trace-children=yes to EXTRA_TEST_ARGS if you want Valgrind to trace into the OOPP helper process(es).

As of December 2014, using the tips in the next section, it is possible to do a complete run of mochitests-plain on Valgrind in about 8 CPU hours on a Core i4910 (Haswell) machine.  Maximum process size is 5.4G, of which about 80% is in memory.  Runs of small subsets of mochitests take far less memory.  This level of resource usage puts it within easy reach of automated nightly testing on individual developer machines.

Tips for improving performance and accuracy of Valgrind's Memcheck tool

Running Firefox on Valgrind's Memcheck tool can be a frustratingly slow experience.  But there are things you can do to improve this.  None of the following is by itself a silver bullet, but taken together they do help considerably.  A sample mozconfig file incorporating all the suggestions is shown below.

(1) MANDATORY: build Firefox with JEMalloc disabled ("ac_add_options --disable-jemalloc").  Despite considerable efforts, Memcheck does not adequately understand JEMalloc's behaviour, and you will be swamped by false error reports if you use it.  Also, you will lose most of Memcheck's error detection capabilities.  Hence it is pretty much mandatory to use the standard system implementation of malloc/free/new/delete if you want sensible results.

(2) MANDATORY: build Firefox with Valgrind hints enabled ("ac_add_options --enable-valgrind").  If you don't do this, Memcheck will go nuts when it sees the JS engine's garbage collector scanning the thread stacks.

(3) MANDATORY: run Valgrind with --smc-check=all-non-file, always.  Otherwise it will crash in JIT-created code.  Also, some of the MacOSX system libraries generate code on the fly, and will not run correctly without this flag.  On ARM CPUs you can omit it, since ARM requires explicit icache invalidation and Valgrind observes and honours such requests.

(4) MANDATORY: run Valgrind with --vex-iropt-register-updates=allregs-at-mem-access, always.  Otherwise it will crash in OdinMonkey-created code.

(5) MANDATORY: run Valgrind with --show-mismatched-frees=no. Due to differential inlining of new vs delete -- that is, new gets inlined but delete doesn't, or vice versa -- Valgrind will often show false mismatched-free errors, which are time consuming to investigate.  Until we arrive at a better solution, the simple solution is to use this flag to hide them.  If you are sure you have a setup that doesn't suffer from differential inlining, you can omit the flag.

(6) MANDATORY for MacOS: run Valgrind with --dsymutil=yes, always.  Otherwise you won't get line number information in error messages.  See also (12) below.

(7) MANDATORY for Linux: run with environment variable "G_SLICE=always-malloc".  This is necessary to get the Gnome system libraries (I think) to use plain malloc, instead of pool allocators.

The rest of the points are not mandatory, but you'd nevertheless be wise to pay attention to them.

(6) Use a decent machine.  Valgrind is tremendously memory-intensive, so the single most important factor is having a large level 2 cache.  2MB is a bare minimum, 4MB or more is preferable.  Most mid-to-upper range Intel Core 2s and Core iXs have 4MB or larger L2/L3s and work well.  I'd guess the latest mid-to-upper-range AMDs are also good, although I haven't tried them recently.  RAM is also important: 4GB of RAM is a bare minimum, and 8GB is better.

(7)  Build Firefox with "-g -O".  Don't use a plain "-g" (unoptimized) build.  Checking memory references takes Valgrind a lot of time.  At -O0 (no optimization), gcc does't do much register allocation, so the generated code has many unnecessary memory references which slow Valgrind down.  At -O (that is, -O1) most of those disappear, whilst retaining pretty good stack-unwind-ability, so that Valgrind can still produce sane stack traces.  The difficulties with running optimised code on Memcheck are a somewhat increased risk of false positive uninitialised-value errors, and incomplete or incomprehensible stack traces.  At -O1 neither of these seem significant.  If you want to live dangerously, and you have gcc-4.3 or later, try "-g -O2".

(8) Use 64-bit builds of Firefox in preference to 32-bit builds.  64-bit code has more available registers and better calling conventions, both of which reduce the number of memory references Valgrind has to check.  Also, the overhead for --smc-check=all-non-file is lower on 64-bit targets.

(9) Don't use --track-origins=yes unless you are hunting down a specific uninitialised-value error.  It pretty much halves the speed of Valgrind.  That said, it is still way faster than tracking down sources of uninitialised data by hand, and so constitutes a net programmer productivity win.

(10) You can make stack traces easier to read by asking for source file names to be given relative to the root of your source tree.  Do this by using --fullpath-after= to specify the rightmost part of the absolute path that you don't want to see.  For example, if your source tree is rooted at /home/sewardj/MC-20-12-2014, use --fullpath-after=2014/ to get path names relative to the source directory.

(11) Use Linux rather than MacOS.  Unfortunately, Valgrind has difficulties in with threaded code on MacOS, which sometimes cause it to run far slower than on Linux.  Fixing this in Valgrind will not be simple.  Such difficulties do not occur on Linux.  If your code is single threaded you can of course ignore this point.  Bear in mind that a startup of the browser plus a bit of surfing causes easily a couple of dozen threads to get created, so all browser startups will encounter this problem on MacOS.

(12) Use the latest Valgrind trunk from SVN.  It's easy to download and build.  The trunk sometimes contains optimizations and bug fixes not yet present in formal releases.  At a bare mininum, use the stock 3.10.1 rather than ancient versions (3.6.x, 3.7.x, etc).

(13) If you get swamped by large numbers of undefined-value or invalid-address errors, first check that your Firefox and Valgrind setup is as this document describes.  The reason is that the recent Firefox trunk is very Valgrind-clean, and most properly configured Valgrind runs on desktop Linux find few (or, often, zero) genuine errors.  Finding a lot of errors is almost always a sign of your setup not being in accordance with this document.

Using all these together, on a Core i5 670 (3.46 GHz) running 64-bit Linux, I can surf the web, reading news sites over morning coffee, whilst running on Memcheck.  The delays are such that it is obvious that Fx is not running natively, but they are small enough that I spend most of my time reading and not much time waiting for Fx.  Here's a recommended mozconfig:

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/ff-opt
ac_add_options --enable-tests
ac_add_options --enable-optimize="-g -O2"
ac_add_options --disable-jemalloc
ac_add_options --enable-valgrind
mk_add_options MOZ_MAKE_FLAGS="-j4"

As per comments above, for more modern Valgrind versions (3.9.x and later) and gcc versions (4.7.x and later) you can use -O2 rather than -O, for marginally improved performance.

Per-platform comments (updated 13 Jan 2015)

Linux on X86_64/X86/ARMv7/AARCH64/PPC32/PPC64/MIPS32/MIPS64/S390X

These work out of the box, either via the 3.10.1 release or from trunk sources.  The X86_64, X86 and ARMv7 ports are widely used for working with Mozilla sources.  The other targets should in principle be able to run Firefox. 

On platforms other than X86_64, X86 and S390X, explicit icache invalidation for JIT generated code is mandatory.  Valgrind observes and honours the such directives, so you get transparent support for JIT-generated code without having to use --smc-check=all-non-file.  No such luck on X86_64, X86 or S390X, unfortunately.

On ARMv7, Valgrind will attempt to use EXIDX unwind information if it is available.  This is useful for unwinding through system libraries on Android and Firefox OS.  You should still try to use -g for Gecko code, though.  This doesn't mean you can't use optimisation -- the standard recommendations of "-g -O" or "-g -O2" still apply.

For un-released Linux distros (Fedora Rawhide, etc) you'll need the trunk sources, since fixes for the latest gcc and glibc versions appear there first.  Without them you'll be flooded with false errors from Memcheck, and have debuginfo reading problems.

Mac OS X 10.10 (Yosemite)

Valgrind trunk works out of the box on these platforms, but there are a number of important extra caveats to be aware of.

  • Don't use a tarball build (3.10.x) since it doesn't contain Yosemite support.
  • You need run permanently with --dsymutil=yes, otherwise you'll never get any line number info.
  • Only 64 bit processes are supported.  32 bit processes kind-of work, but not well enough to run a 32 bit build of Firefox.
  • See caveat (11) above regarding performance on heavily threaded code (like, for example, Firefox.)
  • Valgrind has been observed to cause stability problems with the MacOS kernel -- in short, the kernel may panic and restart the machine.  The reasons for this are unknown.
  • Expect lower performance and a somewhat higher false error rate than on Linux.
  • Valgrind's handling of malloc zones on Yosemite isn't very good, and can lead to false reports of memory leaks.  Regard leak reports with suspicion, therefore.  This could be improved with a couple of days of work.
  • Mac OS X 10.9 (64-bit) might work -- the work to improve 10.10 also improved 10.9 -- but given that Valgrind development resources are limited, 10.10 (64-bit) is the primary support target on Mac OS.

 

Document Tags and Contributors

Last updated by: nnethercote,