Distributing builds within Mozilla offices using IceCC is soon to be deprecated. Please refer to documentation for sccache-dist if distributing from within an office.
Icecream (IceCC) is a tool used to create a pool of machines to distribute compilation initiated on one machine across other machines with idle cycles in the pool. It can significantly speed up your macOS or Linux builds.
DistCC vs. IceCC
Both DistCC and IceCC are distributed compilers. They require similar tool chain on all target machines. However, IceCC has a built-in mechanism for distributing a custom toolchain on demand (in a chroot jail), which means that unlike DistCC (which requires careful management and setup of a toolchain) with IceCC each client can use their own toolchain without any setup. IceCC also has a scheduler so that it will only use idle cylces on other machines in the pool, avoiding overloading other people's machines.
IceCC on Linux
The following instructions are for setting up IceCC on Linux:
Debian or Ubuntu
sudo apt install icecc
This will automatically start an IceCC Daemon, which should be able to automatically locate and connect to the scheduler running on your network using UDP multi-cast.
Note that Debian/Ubuntu prior to 2018 ship icecream 1.0.1 which has some stability issues. We recommend upgrading to 1.1.
sudo dnf install icecream sudo systemctl enable iceccd sudo systemctl start iceccd
# Add these lines to your mozconfig to point it at the compiler wrappers CC="/usr/lib/icecc/bin/cc" CXX="/usr/lib/icecc/bin/c++"
IceCC Client on Mac
The following instructions are for setting up IceCC as a client on macOS. Your macOS machine will not service requests, but will distribute jobs to any x86-64 machines on the network (linux machines), and use them to compile, in addition to local compiles.
This will involve using a custom toolchain which supports cross compilation. This toolchain has been bundled for ease of use.
xcode-select --install brew install lzo
Download and a Cross-Compilation Toolchain
You need two builds of the toolchain. One build for the remote Linux host(s) in the build pool to which compilation will be distributed, and one build for the local macOS machine. These toolchain builds must match.
jya made toolchain bundles available up until clang 7.0. For more recent clang toolchains you will currently need to create your own.
The older bundles come with an install script:
git clone https://github.com/mstange/icecc-osx-moztor ~/icecream
Unfortunately, from my testing, the icecream daemon on macOS has trouble discovering schedulers with UDP broadcast, so we explicitly specify the scheduler to connect to when installing the daemon. Install the daemon as follows. If you don't know your scheduler IP, you can check the table at the bottom of this document.
sudo ~/icecream/install.sh SCHEDULER_IP_ADDRESS
# Add these lines to your mozconfig to point it at the compiler wrappers CC="$HOME/icecream/cc" CXX="$HOME/icecream/c++"
By default, with these build setups, jobs will not be run on mac machines in your network. Mac machines which want to run jobs will have to create an `icecc` account on their computer, and computers which want to run jobs on them will have to distribute a cross compilation toolchain. In the Toronto office, we decided to not bother with running jobs on macOS machines.
You can use ccache together with icecc.
On Fedora, no additional configuration should be necessary (since ccache is set up to work transparently out of the box).
On Debian or Ubuntu, because your mozconfig file gets sourced by configure and client.mk, you need to save the variable during configure to expose it to the make environment:
mk_add_options 'export CCACHE_PREFIX=icecc' export CC=clang export CXX=clang++ ac_add_options --with-ccache
Note that when using ccache (which you should!) you don’t want to include icecc in the CC or CXX output variables. As we want ccache to be the primary entry-point, we set a prefix to the command line that ccache uses when invoking the compiler.
These instructions won't work with the OSX instructions above (You will likely get errors related to ICECC_VERSION not being set correctly). Instructions for how to use ccache with OS X are forthcoming.
You may need to manually clobber your build directory for mach to detect that you're now using icecream:
You will want to run future builds with enough jobs to saturate your cluster. For example:
./mach build -j 100
You can also specify this flag in your mozconfig, to avoid typing it after every `build` command:
You can also use a script like https://gist.github.com/mystor/27beaf86a4868a357dcd58f585b42c6a to get the number of jobs in your network, and set the MOZ_MAKE_FLAGS that way.
At least one machine per network will need to run the IceCream scheduler (icecc-scheduler) program. These computers will be responsible for scheduling jobs in the network and dispatching them to computers to be executed. The following are the addresses of the schedulers in each office:
|Office||Hostname or IP Address of Scheduler|
Do not use a scheduler for a different office than your own! The latency of this connection will make everyone's builds slower.
In general, avoid connecting to the network except over wired LAN, for minimal latency and the best performance.
The Icecream monitor
There is a monitor program, icemon which can be run to give a visual interface to the activity currently in the IceCream network. It has some useful, and nice to look at, views, such as the Gnatt view, which provides a satisfying display of the current jobs running on the network.
On some platforms, such as macOS, icemon can't discover the scheduler automatically. If this is the case, use the environment variable USE_SCHEDULER=SCHEDULER_IP.
Frequent problems & debugging
Many sporadic errors can be fixed by simply restarting your local daemon. Here's the command to do that, on Ubuntu:
sudo service iceccd restart
For more details & specific known issues, read on.
Configure error: CoreMedia/CoreMedia.h file not found
This happens on macOS when the SDK headers are not installed (building without icecc, this issue would typically manifest later in the build process as an inttypes.h file not found error). Run
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg to install the SDK headers.
It’s not working, how do I debug this?
Start all processes with
-vvv to see tracing information.
My machine shows up in icemon but does not accept jobs from other nodes
Your daemon version might be too old, or the client might not have a suitable toolchain to use that node. Or, you may need to open TCP port 10245 in your firewall (check to be sure you can telnet to that port from another machine).
My machine shows up in icemon but does not distribute jobs to other nodes
You're likely using a version-specific compiler binary, like "clang++-3.6" or "gcc-4.8". Unfortunately, these filenames confuse icecc, due to a bug -- for best results, only use compiler binaries that are named clang/clang++/gcc/g++ in your CC/CXX environmental variables.
Everything is really slow!
You probably either don't have your icecream daemon running (check if a process named iceccd is running), aren't connected to the scheduler, or are not on a LAN wired connection. A good connection to the network is essential for icecream performance.
Another source of slowness is that sccache causes the number of build jobs to be limited to the available local cores, despite setting
mk_add_options MOZ_MAKE_FLAGS="-j100" or using
./mach build -j100. With icecream, it is best to avoid sccache.
Some jobs fail with exception 17
In pre v1.1 releases (you are probably using v1.0.1), if the builder runs out of memory, error/exception 17 is thrown without explanation. On more recent releases, the error message will explicitly say "Error 101 - the server ran out of memory, recompiling locally". If you can make more memory available on the builder (ex: if it's a VM), do that. Otherwise, you probably want to reduce the ICECC_MAX_JOBS. In Ubuntu, the configuration value can be found in /etc/icecc/icecc.conf.
All out-going jobs failed with
This means that your machine makes wrong toolchain. Remove a few Ubuntu packages will fix this issue.
sudo apt purge hardening-wrapper hardening-includes
I get build failures due to -Werror when building remotely but not when building locally
Icecream expands all source with the preprocessor before sending it over the wire, so any warnings which are silenced when they are produced by code within macros won't be silenced when building on the remote machine. Try building without --enable-warnings-as-errors to get the build to succeed over icecream.
Alternatively, you could keep --enable-warnings-as-errors, and just disable certain problematic warnings. I found disabling -Wunreachable-code, -Wparentheses-equality, -Wundefined-bool-conversion, -Wunused-value, -Wconstant-logical-operand, -Wtautological-compare, and -Wformat-extra-args to be sufficient.
Debug information in Mac is missing (stack traces are wrong)
This problem is because Mac icecream is using clang, and clang needs a different command line argument (-fdebug-compilation-dir) to know where the file it's compiling is located, so that it can write out correct file paths in the debug info. Icecream is currently not passing this command line argument to the compiler. We could either fix icecream to pass it (mstange and mystor tried to write a patch to do this but didn't succeed immediately and gave up), or we could change the Firefox build system to pass that option. At the moment we don't have a solution.
I get errors like "invalid conversion" or "use of undeclared identifier"
Typical errors look like:
.mozbuild/clang/lib/clang/8.0.0/include/avx512bwintrin.h:634:10: error: invalid conversion between vector type '__m512i' (vector of 8 'long long' values) and integer type 'int' of different size
.mozbuild/clang/lib/clang/8.0.0/include/bmiintrin.h:199:10: error: use of undeclared identifier '__builtin_ia32_tzcnt_u32'
This happens because you use icecc 1.2 or older, that can't figure custom clang installations. You should install a newer version of icecc (either 1.3 if it's out yet, or build a master version from their repository).
I get errors like "flush_writebuf() failed(Error: Broken pipe)"
ICECC_DEBUG=debug, you get a more specific error like:
ICECC 2019-09-11 11:44:22: daemon can't determine native environment. Set $ICECC_VERSION to an icecc environment.
The reason could be that the clang in
.mozbuild isn't reachable by your iceccd daemon. Possibly the
.mozbuild directory do not have the necessary rights: it should at least have the
x mode for
Scratch notes of how the toolchain was generated. These are a bit out date against what we exactly host (builts against trunk and not 3.8). But it should be easy to get running from these instructions.