Find errors in packages through mass builds

Even after thorough unit testing on both applications and their library dependencies, builds often go wrong and turn up hidden errors. This article introduces a new tool, the Mass Prebuilder (MPB), that automates builds on enormous numbers of reverse dependencies to find problems that are not caught through package testing.

Let's look at a simple example. Roughly 1,200 packages in Red Hat-based distributions depend on GNU Autoconf. Knowing all of the packages by heart is unlikely, building them manually would take ages, and judging whether a failure is due to a change in GNU Autoconf is very difficult. A job for the Mass Prebuilder!

What is the Mass Prebuilder?

The Mass Prebuilder is an open source set of tools that help developers create mass rebuilds around a limited set of packages, in order to assess the stability of a given update.

The idea is rather simple. The packages your team is working on and want to release are the engineering packages. Given a package or a set of packages, called the main packages, the Mass Prebuilder calculates the list of their direct reverse dependencies: packages that explicitly mark one of the main packages in their BuildRequires field.

The Mass Prebuilder first builds the main packages using the distribution's facilities, which should include a set of test cases that validate general functioning. Assuming these packages are built successfully, they are then used as base packages to build the reverse dependencies and execute their own test cases.

This process yields a first set of results, which include successful builds (hopefully the majority), but also failures that might or might not be the result of changes introduced by modifications to the main packages. To clarify the source of the problem, as soon as a failure is detected, the Mass Prebuilder creates another mass build that includes only the reverse dependencies that failed to build during the original run. This new build runs in parallel to the original, but without the changes that were introduced into the main packages—in other words, it's a pristine build.

Once all the package builds are done, they can be broken down into the following categories:

1. Successfully built packages
2. Packages that failed to build only with the modified packages
3. Packages that failed to build with both the modifications and the pristine version

The first category can likely be ignored, because it consists of packages that don't seem to have been affected by the changes.

The second category needs much more attention. These are the ones that cry, "Hey, there seems to be a big problem with your changes." The failures need to be analyzed to figure out the root cause of the problem—they could arise from changes that have been introduced, for instance, or maybe from a mistake made by the final user (e.g., use of a deprecated feature that got removed).

The last category of failures is a bit trickier. Since the build failed with the pristine packages, there may be hidden errors in them that were revealed by the new changes.

How does the Mass Prebuilder work?

Now let's see a bit more in detail what the Mass Prebuilder does under the hood.

The MPB is a set of Python scripts that abstract the use of commonly available infrastructures. Although the infrastructure used initially is Copr, there are plans to implement support for other infrastructures such as Koji and potentially Beaker.

If these infrastructures are used to build the packages, the MPB is in charge of orchestrating the builds by preparing the project, calculating the list of packages to be built, and providing a simple report once everything is done.

Figure 1 shows a simplified overview of the system.

Under the hood, the tool has a back-end/front-end design. Most of the work is done as generically as possible, so that the back ends implement only the direct interfaces with the infrastructures.

The Mass Prebuilder process

The MPB set of scripts is currently available in version 0.2.0, which allows you to:

• Create a dedicated Copr project where an engineering version of your packages can be built. This package can be provided as a source RPM (SRPM), a DistGit repository with a specific tag, a Git repository, or a URL providing access to an SRPM.
• Trigger the build for these packages, and report its success or failure.
• Automatically calculate direct reverse dependencies for the main packages, try to group them by priority based on their interdependencies, and trigger the builds for these reverse dependencies.
• Generate a simple report on the build status for the reverse dependencies. If a reverse dependency isn't building, that triggers a new build on a pristine Copr project (without your engineering packages), and checks to see whether the packages fail there too.

As explained earlier, failures are then split into the following categories:

• Failed: Packages that only failed in the project that included your engineering packages
• Manual confirmation needed: Packages that failed on both sides

At the end of the process, the tool retrieves all the available data for the packages that failed so that you can walk through them and verify whether failures are related to your package update.

The process is therefore decomposed into multiple stages:

• Preparation (your engineering packages are built)
• Checking the preparation
• Build
• Checking the build
• Collecting and reporting data

The process can be interrupted safely during the checks and the collecting operations. If you interrupt the Mass Prebuilder, it shows you the command to restart it later on. This checkpointing is quite useful, as some packages take a long time to build and you might not want your machine running for 30 hours straight.

Configuration

Prior to using the Mass Prebuilder, make sure you can communicate with the appropriate infrastructure. Prepare a small configuration file specifying what you want to build and how. Copr itself requires a ~/.config/copr file (as given by the Copr API when logged in) to ensure that the copr whoami command gives the expected output.

The Mass Prebuilder configuration file can be provided in the following ways. It looks for the files in the order shown, and uses the first file it finds:

1. A local mpb.config file

2. Through a command-line option:

   $ mpb --config=~/work/mpb/autoconf/my_config_file

3. In the default path: ~/.mpb/config

The following example uses a local file to show how to use the tool to check your use of the Autoconf:

$ mkdir -p ~/work/mpb/autoconf
$ cd ~/work/mpb/autoconf
$ cat > mpb1.config << EOF
> arch: x86_64
chroot: fedora-rawhide
packages:
  autoconf:
    src_type: file
    src: /home/fberat/work/fedora/autoconf/autoconf-2.72c-1.fc37.src.rpm
data: /home/fberat/work/mpb/autoconf
verbose: 1
> EOF
$ mpb
Loading mpb.config
Using copr back-end
Populating package list with autoconf
Executing stage 0 (prepare)
Prepared build mpb.18 (ID: 18)
Executing stage 1 (check_prepare)
Checking build for mpb.18 (ID: 18)
You can now safely interrupt this stage
Restart it later using one of the following commands:
"mpb --buildid 18"
"mpb --buildid 18 --stage 1"
Build status: /
        0 out of 1 builds are done.
        Pending: 0
        Running: 1
        Success: 0
        Under check: 0
        Manual confirmation needed: 0
        Failed: 0

The configuration file can contain a lot of parameters. We'll focus here on name and reversedeps.

The name configuration parameter

You can specify a name that will help you identify your project. This name will replace mpb.18 from the previous example. Be careful, though, because this parameter will also be the name used in your Copr project. If the name already exists, specifying it might lead to unexpected behavior. If you choose to set the name yourself instead of having it automatically generated, I recommend adding a line to the configuration file as follows. Replace <N> with the number given by the build, which was 18 in the previous example:

$ echo "build_id: <N>" >> mpb.config

Restart the MPB later using either of the following commands:

$ mpb --buildid <N>
$ mpb --buildid <N> --stage 1
$ mpb # If you stored the build_id in the configuration file

The reversedeps configuration parameter

The reversedeps parameter contains a list of reverse dependencies. If you specify this parameter, the Mass Prebuilder uses your list instead of calculating the reverse dependencies by checking the main packages. This parameter is useful if you want to rebuild only a subset of packages instead of, say, the 6,000+ ones for GCC.

An example using reversedeps follows:

> arch: x86_64
chroot: fedora-rawhide
packages:
  autoconf:
    src_type: file
    src: /home/fberat/work/fedora/autoconf/autoconf-2.72c-1.fc37.src.rpm
reversedeps:
  list:
    libtool:
      priority: 0
    automake:
      priority: 1
name: autoconf272-1
data: /home/fberat/work/mpb/autoconf
verbose: 1
> EOF

You can include the priority field in the reverse dependencies to specify a build order. The previous example builds libtool before automake. If you don't need to control the build order, the previous configuration can be simplified to:

  reversedeps:
     list: libtool automake

Getting results from the Mass Prebuilder

Let's come back to our Autoconf build. After a while (about 30 minutes during one test run), the tool can move to the next stage and calculate the reverse dependencies that would be valid for an x86-64 processor:

    Executing stage 2 (build)
    Calculating reverse dependencies.
    Level 0 depth for x86_64
    Checking 2316 packages.
    Retrieved 1151 packages.
    Prepare discriminator for priorities calculation
    100% done
    Setting priorities
    7 pass done.
    Populating package list with [package names here]

In this case, 1,151 packages are built for x86-64. The build stage is followed by a check_build stage:

    Executing stage 3 (check_build)
    Checking build for autoconf272-1 (ID: 18)
    You can now safely interrupt this stage
    Restart it later using one of the following commands:
    "mpb --buildid 18"
    "mpb --buildid 18 --stage 3"
    Build status: \
    3 out of 1151 builds are done.
            Pending: 1146
            Running: 1
            Success: 3
            Under check: 1
            Manual confirmation needed: 0
            Failed: 0

After all these builds finish, the Mass Prebuilder collects data that you can retrieve from the following path, based on your configuration:

<data field>/<name of the MPB project>/<build status>/<chroot>/<name of the package>

For my particular run, the paths are:

~/work/mpb/autoconf/autoconf272-1/FAILED/fedora-rawhide-x86_64/
~/work/mpb/autoconf/autoconf272-1/UNCONFIRMED/fedora-rawhide-x86_64/

A real-life example: Errors building with autoconf2.72c

As of June 2022, GNU Autoconf 2.72 is still under development and is therefore unstable. Yet the upstream maintainers are tagging the mainline with intermediate versions that could be useful to test early, in order to limit the problems when the final release comes out.

This is where the MPB comes in handy.

In Fedora 36 x86_64, about 1,151 packages depend directly on Autoconf. When built with Autoconf 2.72c pre-release, we currently get the following result:

• Success: 1,033
• Manual confirmation needed: 47
• Failed: 71

Let's skip the "Manual confirmation needed" packages, which failed to build with both the 2.72c version and the original 2.71 version. There may be failures to fix among them, but the "Failed" packages offer enough interesting examples for this article to keep us busy.

A common pattern turns up in the 71 failures: 65 of them are due to a malformed configuration script. A sample error found for the PHP package is:

./configure: line 104153: syntax error: unexpected end of file

Although this seems to be quite a common failure, it went through the internal tests from Autoconf without being noticed.

Six more failures need deeper analysis:

• am-utils: A straightforward failure due to a hard requirement for the Autoconf version (it requires 2.69 or 2.71 exclusively).
• cyrus-imapd: A missing krb.h header. Even though the configure stage reports that the file isn't available, the application still tries to use it. It's strange not to find this header, and the message may hide a bigger problem.
• libmng: A zlib library not found error because zlib-devel got installed as part of the dependencies.
• libverto: This failure seems unrelated to Autoconf; one of its internal libraries seems to have changed its name. Yet it is strange that this failure appeared only with Autoconf 2.72c.
• mingw-libmng: Unresolved symbols during linking. May be unrelated to Autoconf, unless a change in the configuration modified the build process.
• nfdump: An error revealed during the configure run; the FT2NFDUMP conditional was never defined. This error might also be due to a malformed configure script.

The next steps, which are beyond the scope of this article, are to research the errors to gain a deeper understanding of the problems and create appropriate bug tickets for the corresponding components. Although in the current case most of the problems are likely in Autoconf itself, there may be some cases where the issues are in the user's component (such as am-utils here).

Overall, this sample run of the Mass Prebuilder gives a good idea of the kind of failures that would have been missed by simply building the Autoconf package and relying on its internal tests for gatekeeping. The large panel view provided by running with a distribution is quite beneficial, and can improve the overall quality of the packages being provided.

Where can you find the Mass Prebuilder?

As of July 2022, the tool is available as a package at my Copr repository. Packages are available for Extra Packages for Enterprise Linux (EPEL) 9 and 8, and for Fedora 35, 36, and 37.

The sources can be found in my GitLab repository.

Current limitations

The Mass Prebuilder at the time of writing is still under heavy development. That means that there might be missing features you'd like to see, that there might be bugs (though I've done my best to limit them), and that interfaces may have to change a bit by the time stable releases come.

While I tested the tool, it appeared that builds are not fully reliable. Although reported successes are trustworthy, reported failures might not be. There were some cases where the failure cropped up because the infrastructure was incapable of installing build dependencies, even on a stable release such as Fedora 35. This failure is hard to diagnose because it doesn't make sense. The same build with no changes could result in a different status if started with a few seconds delay.

The data that can be collected out of a failed Copr build is relatively limited. For instance, in the problem just described, there is no way to retrieve the failing configuration script. A local build needs to be made for that.

If you have any suggestions, if you find an issue, or if the tool doesn't behave the way you expect, don't hesitate to contact me and give me feedback, or leave a comment at the bottom of the article.