Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

2 tips to make your C++ projects compile 3 times faster

May 15, 2019
Konrad Kleine
Related topics:
ContainersLinux
Related products:
Developer Tools

Share:

    In this article, I will demonstrate how to speed up your compilation times by distributing compilation load using a distcc server container.  Specifically, I'll show how to set up and use containers running a distcc server to distribute the compilation load over a heterogeneous cluster of nodes (development laptop, old desktop PC, and a Mac). To improve the speed of recompilation, I will use ccache.

    What to expect

    Today, you can find a lot articles about Kubernetes or OpenShift. This is not such an article. I started using containers, especially Docker, back in 2013, and in the beginning, I mainly used them to isolate things and have reproducible builds. The same is true here; we'll build a container, push it to DockerHub for distribution, and download it on other machines on the fly when running docker run ... so don’t expect crazy things to happen here.

    In fact, even if you don’t have any prior knowledge of Docker, you should still be able to follow along. Getting and installing the Docker Engine in the Community Version (on Linux) or the Docker Desktop (on macOS and Windows) is completely enough.

    In terms of C++, I don’t expect anything code-wise except that your project uses CMake and compiles with GCC or Clang.

    Introduction

    I recently started a new job at Red Hat where I get the chance to work on the LLDB project. The first thing I noticed, after not having worked with C++ for quite a while, was the slowness of compilation. I worked with Go for a couple of years, and compilation performance isn’t an issue there at all. It simply vanishes from the picture.

    Slow compilation

    When I say that compilation was slow for LLDB, I mean that compiling the lldb target (git tag: llvmorg-7.1.0) in release mode takes close to two hours. Release builds are already the fastest to build and given that I need to build a total of 7 or more variants of LLDB (clang/gcc, debug/release, asan/no asan, assert/no assert, …) this makes compilation time take up a whole working day.

    My normal developer system consists of a three-year-old Lenovo Thinkpad T460s laptop, on which I run Fedora 29 and the following tools:

    • gcc (GCC) 8.2.1 20180801 (Red Hat 8.2.1–2)
    • clang version 7.0.1 (Fedora 7.0.1–6.fc29)
    • ccache version 3.4.2
    • distcc 3.2rc1 x86_64-redhat-linux-gnu

    When I compile LLDB on my development laptop using make and four compile jobs ( -j 4 ), the time command reports this:

    real    72m24,439s
    user    224m32,731s
    sys     12m2,094s

    This means that I have to wait approximately two hours for a one-time compilation.

    Speeding up

    When I compile LLDB on my cluster (laptop, desktop, iMac) using ninja and 10 compile jobs (as determined by distcc -j ) the time command reports this:

    real    22m29,130s
    user    49m16,726s
    sys     4m27,407s

    That means compiling with ninja and my distcc cluster is

    • 3.22 times faster in real time.
    • 4.56 times faster in total number of CPU seconds that the process spent in user mode.
    • 2.7 times faster in number of CPU-seconds that the process spent in kernel mode.

    I hope this motivates you to continue reading.

    distcc

    Tip #1: Distributing compilation load

    I have experimented before with ways to speed up compilation using distcc, which lets you distribute your compile jobs onto other machines. It requires you to have the exact same compilers installed on the worker machines (the servers) as on the developer machine (the client).

    So I installed a spare machine with Fedora 29 running on it to make it part of my compilation cluster. I installed all the right compilers in the right versions to match the ones I have running on my local machine.

    But then I realized that this would be a nightmare to maintain. I didn’t want to trade slowness for complexity. Not to mention all the things I would have to do when I upgrade my developer system to Fedora 30. Do I really have to upgrade all my worker machines in the cluster?

    Also, I had an iMac sitting on another desk that doesn’t do much when we’re not doing video or photo editing or music recording with it. Could it be used for compilation? After all, macOS doesn’t have the exact same compilers that I have on my developer machine. This is what I meant by heterogeneous cluster of nodes in the introduction.

    To spare you the details of how to set up a distcc server, or how to bring CMake to use it, we’ll jump into Tip #2 right away.

    container

     

    Tip #2: Using a distcc server container

    I’ve mentioned before that I have an iMac, which does nothing when I work on my laptop. Wouldn’t it be nice if I could run a container (e.g., with Docker) on it that serves distcc with all the necessary tooling already set up?

    To better explain what I'm trying to achieve, here’s a diagram of my compiler cluster architecture.

    Diagram of my compiler cluster architecture

    Diagram of my compiler cluster architecture.">

    It turns out, you can (almost) completely abstract away from the underlying operating system as long as you can run a docker run command in a terminal.

    If you’re not familiar with Docker, don’t worry; it is easy. Think of this rough analogy:

    • A Docker image is like a program in Linux.
    • A container is like a process of a program in Linux, a.k.a. an instance of the program.

    Obviously, the image must exist before the container can exist. To create an image, you write what’s called a Dockerfile that describes the base image  (think operating system here, even though it is not correct) you want to begin with. In my case, I want to be as close to my developer machine as possible, so I use the fedora:29 image.

    FROM fedora:29

    Then, you install tools that your image ships with (installing tools):

    RUN dnf install -y \
        clang \
        distcc \
        distcc-server \
        doxygen \
        gcc \
        graphviz \
        htop \
        libasan \
        libasan-static \
        libedit-devel \
        libxml2-devel \
        make \
        ncurses-devel \
        net-tools \
        python-devel \
        swig \
       && yum clean all

    NOTE: All of the above packages are not necessarily needed. I literally installed in my Docker image whatever I had installed on my developer laptop. This, of course, is not needed because distcc only sees pre-processed compilation units. That means it does not need to have all the include files available itself, but instead they will be pushed to the distcc server from my developer laptop.

    In a Docker container, there can only be one top-level process that is the heart of the container. In our case, that is distcc. We configure it using a so-called entrypoint and pass along all the flags that we want to keep independent of the machine that runs the container:

    ENTRYPOINT [\
      "distccd", \
      "--daemon", \
      "--no-detach", \
      "--user", "distcc", \
      "--port", "3632", \
      "--stats", \
      "--stats-port", "3633", \
      "--log-stderr", \
      "--listen", "0.0.0.0"\
    ]

    If you don’t understand any of the flags above, you can look them up with man distccd.

    Then, there are the flags for which we provide defaults but that you also can change:

    # By default the distcc server will accept clients from everywhere.
    # Feel free to run the docker image with different values for the
    # following params.
    CMD [\
      "--allow", "0.0.0.0/0", \
      "--nice", "5", \
      "--jobs", "5" \
    ]

    That is essentially it. You can find the latest version of my Dockerfile here.

    How to build the Docker image out of the Dockerfile

    Please note, that I’ve configured a repository on DockerHub to automatically build the latest version of my Dockerfile image. Instead of building the image yourself, you could just issue the following command to get the latest version:

    $ docker pull konradkleine/distcc:fedora29

    If you plan to experiment and tinker with it, you can build the image like so:

    $ git clone git@github.com:kwk/distcc-docker-images.git
    $ cd distcc-docker-images
    $ docker build -t konradkleine/distcc:fedora29 -f Dockerfile.fedora29 .
    

    How to run the Docker image

    To make a Linux, MacOS or Windows-based computer part of your cluster, make sure you have installed and properly configured Docker on that machine. Then head over to a terminal and run the following command:

    $ docker run \
      -p 3632:3632 \
      -p 3633:3633 \
      -d \
      konradkleine/distcc:fedora29

    This will download (a.k.a. pull) my latest distcc Docker image (if not already pulled) and run it in daemon mode ( -d ). It will expose distcc’s main port ( 3632 ) and distcc’s HTTP statistics port ( 3633 ) under the same numbers on the host machine.

    How to test the Docker image

    The beauty of Docker to me is that you can always use your developer machine to try out stuff. For example, to run the distcc container on your localhost, you can run the same command from before but give the container a name this time so you can refer to it in subsequent Docker commands:

    $ docker run \
      -p 3632:3632 \
      -p 3633:3633 \
      -d \
      --name localdistcc \
      konradkleine/distcc:fedora29

    Then I suggest running htopinside the newly created container to see what’s going on:

    $ docker exec -it localdistcc htop
    
    Bildschirmfoto von - Running htop inside the localdistcc container shows

    Running htop inside the localdistcc container shows that your container only runs distccd.">

    Let’s compile some code and distribute it to our localdistcc node. Here I’m going to compile LLDB with adjustments:

    $ git clone https://github.com/llvm/llvm-project.git ~/dev/llvm-project
    # Get the IP address of the localdistcc container
    $ export LOCAL_DISTCC_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' localdistcc)
    $ export DISTCC_HOSTS="$LOCAL_DISTCC_IP/7 localhost"
    $ mkdir -p ~/dev/llvm-builds/release-gcc-distcc
    $ cd ~/dev/llvm-builds/release-gcc-distcc
    $ cmake ~/dev/llvm-project/llvm \
      -G Ninja \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_USE_LINKER=gold \
      -DLLVM_ENABLE_PROJECTS="lldb;clang;lld" \
      -DCMAKE_C_COMPILER=/usr/bin/gcc \
      -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
      -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
      -DCMAKE_C_COMPILER_LAUNCHER="ccache;distcc" \
      -DCMAKE_CXX_COMPILER_LAUNCHER="ccache;distcc"
    $ ninja lldb -j $(distcc -j)

    I’ve marked the sections in bold that are worth looking at. Things to notice are:

    Get the IP address of the container running the distcc container using docker inspect .

    When exporting DISTCC_HOSTS, I suggest that you take a look at the /7 because that tells distcc to distribute seven jobs to this machine. My machine has eight cores, and I don’t want it to run out of power. By default, only four jobs are sent (see man distcc):

    /LIMIT A decimal limit can be added to any host specification to restrict the number of jobs that this client will send to the machine. The limit defaults to four per host (two for local‐host), but may be further restricted by the server. You should only need to increase this for servers with more than two processors.

    Generate a build system for ninja using -G Ninja.

    We use fully qualified paths to the compilers we want to use by specifying CMAKE_C_COMPILER=/usr/bin/gcc and CMAKE_CXX_COMPILER=/usr/bin/g++ . See the next section about masquerading to understand why we do this.

    Use CMake’s CMAKE_<LANG>_COMPILER_LAUNCHER in order to call ccache distcc /usr/bin/gcc instead of just /usr/bin/gcc. This is a nice mechanism to avoid masquerading gcc or g++ .

    Masquerading is a technique where you create file named gcc and make it available on your PATH to intercept calls to gcc. Oddly enough, Fedora 29 does this when you install ccache . That package creates /usr/lib64/ccache/gcc that points to /usr/bin/ccache . This effectively masquerades your local gcc binary with ccache . As a matter of fact, ccache will masquerade a whole armada of compilers including but not limited to gcc, g++, clang, clang++ (see rpm -ql ccache for the complete list). This is why we use fully qualified paths to the compiler /usr/bin/gcc .

    We let discc figure out how many build jobs to run in parallel by looking at the available hosts: -j $(distcc -j) .

    While ninja builds, take a look at the distccmon-gnome output (see the "Troubleshooting" section). It should show something like this:

    Bildschirmfoto von

    distccmon-gnome output when compiling locally using a distcc container.">

    NOTE: distcc -j assumes that all your hosts listed in DISTCC_HOSTS are distinct machines. But the container runs on your localhost and thereby consumes resources from it. This could impact the performance of your developer machine. If things get slow, just abort the compilation; this is just for testing anyway.

    Troubleshooting

    How can I visualize the distribution of the compilation?

    On Fedora 29, I use distccmon-gnome or distccmon-text (see screenshots below). To install them, run sudo dnf distcc-gnome distcc .

    How do I test that all my hosts are serving distcc?

    Test that all your nodes are serving distcc with netcat (nc). In the following command, replace 192.168.178.33 192.168.178.39 with the IPs of your hosts. NOTE: I’m often on a VPN, so I use fixed IP addresses to mitigate issues with resolving host names.

    $ for i in 192.168.178.33 192.168.178.39; do nc -zv $i 3632; done
    Ncat: Version 7.70 ( https://nmap.org/ncat )
    Ncat: Connected to 192.168.178.33:3632.
    Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
    Ncat: Version 7.70 ( https://nmap.org/ncat )
    Ncat: Connected to 192.168.178.39:3632.
    Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

    Alternatively, you can use the much slower nmap:

    $ nmap -A 192.168.178.33/32 -p 3632 -Pn
    Starting Nmap 7.70 ( https://nmap.org ) at 2019-04-23 14:05 CEST
    Nmap scan report for MyServer (192.168.178.33)
    Host is up (0.00069s latency).
    PORT     STATE SERVICE VERSION
    3632/tcp open  distccd distccd v1 ((GNU) 8.3.1 20190223 (Red Hat 8.3.1-2))
    Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
    Nmap done: 1 IP address (1 host up) scanned in 6.68 seconds

    For better security, how can I specify who is allowed to use my distcc container?

    Please note, that the CMD section in the Dockerfile defines a default of --allow 0.0.0.0/0 to allow connections from everywhere. I suggest that you adjust this parameter when running the distcc container:

    $ docker run \
      -p 3632:3632 \
      -p 3633:3633 \
      -d \
      konradkleine/distcc:fedora29 \
      --allow <YOUR_HOST>

    Do I have to use ccache or can I just use distcc?

    You can absolutely only use one or the other, just adjust the CMAKE_C_COMPILER_LAUNCHER and CMAKE_CXX_COMPILER_LAUNCHER variables when configuring/generating your project with CMake.

    # To use ccache and distcc
    -DCMAKE_C_COMPILER_LAUNCHER="ccache;distcc" \
    -DCMAKE_CXX_COMPILER_LAUNCHER="ccache;distcc"
    # To use ccache alone (please note, that no distribution will 
    # happen if you choose this option.)
    -DCMAKE_C_COMPILER_LAUNCHER="ccache" \
    -DCMAKE_CXX_COMPILER_LAUNCHER="ccache"
    # To use distcc alone
    -DCMAKE_C_COMPILER_LAUNCHER="distcc" \
    -DCMAKE_CXX_COMPILER_LAUNCHER="distcc"

    How do I send more than 4  jobs to a machine?

    In your DISTCC_HOSTS environment variable, you need to adjust the limit, which defaults to 4. It is specified with a /<LIMIT>after the hostname:

    export DISTCC_HOSTS="fasthost/8 slowhost/2 localhost"

    How can I see if ccache works for me?

    I suggest that you take a look at the output of ccache --show-stats and inspect the cache hit rate. For example, the following hit rate is at 15.61%, which is better than nothing, I’d say.

    $ ccache --show-stats
    cache directory                     /home/kkleine/.ccache
    primary config                      /home/kkleine/.ccache/ccache.conf
    secondary config      (readonly)    /etc/ccache.conf
    stats zero time                     Wed Apr 17 18:44:31 2019
    cache hit (direct)                  1787
    cache hit (preprocessed)              21
    cache miss                          9774
    cache hit rate                     15.61 %
    called for link                      584
    called for preprocessing              35
    compile failed                        12
    preprocessor error                   116
    unsupported code directive             4
    no input file                         19
    cleanups performed                     0
    files in cache                     25218
    cache size                           1.2 GB
    max cache size                       5.0 GB

    I hope you liked reading this article and got some value out of it. Check out the accompanying video, too.

    Last updated: February 6, 2024

    Recent Posts

    • Our top 10 articles of 2025 (so far)

    • The benefits of auto-merging GitHub and GitLab repositories

    • Supercharging AI isolation: microVMs with RamaLama & libkrun

    • Simplify multi-VPC connectivity with amazon.aws 9.0.0

    • How HaProxy router settings affect middleware applications

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue