Best practices for building bootable containers

Image mode for Red Hat Enterprise Linux (RHEL) simplifies the process of building, deploying, and managing Red Hat Enterprise Linux as a bootable container. Development, operations, and solution providers can simply use the same container-native tools and techniques to manage applications, the underlying operating system (OS) and more. 

Please follow our series of articles detailing the experience of using image mode. In these articles, we discussed the various use cases of image mode, creating automated CI/CD pipelines, managing containerized workloads, a full GitOps experience for sysadmins of RHEL, and how image mode facilitates building software appliances.

In this article, I want to talk about best practices for building bootable containers that form the building blocks of image mode for RHEL. 

Building bootable versus application containers

Like ordinary application containers, you can build bootable containers by using existing container technologies such as Containerfiles using existing tooling, such as Podman, Docker or buildkit. You can also store the images on any container registry, such as Quay.io, Docker Hub, the GitHub Container Registry, or any internal container registry. Figure 1 compares application containers and bootable containers.

Bootable containers are a natural evolution of container technologies. For over a decade, containers have evolved into an industry standard of bundling, shipping, and deploying applications. Bootable containers build on top of these existing technologies and extend containers to include the entire operating system and the Linux kernel to allow for a comprehensive container-native workflow and user experience.

Using Containerfiles

Containers are commonly built via Containerfiles, also known as Dockerfiles. Those files include all the information needed to build a container image, such as the base image, instructions to install software packages or to copy files from a Git repository, and much more (Figure 2).

The workflows and tools for building bootable containers are essentially the same as application containers. I see the beauty in that, since we can benefit from and build on top of more than a decade of innovation, rock-solid tooling and community-driven best practices. However, I want to elaborate on several best practices that only apply to building bootable containers. Let’s get started.

Best practices for linting

We recommend running the following bootc container lint command as a final stage in a Containerfile. This command will perform a number of checks inside the container image and throw an error when there is an issue.

FROM quay.io/fedora/fedora-bootc:41
# Customization steps
RUN bootc container lint

Running the bootc linter prevents us from running into certain bugs and helps us keep the image and the content in good shape. The community continuously adds more checks to the command, but the following are the most common ones:

• Check for multiple kernels in /usr/lib/modules. Only one kernel is supported per image, so the command would exit with a failure.
• Make sure that the syntax of files in /usr/lib/bootc/kargs.d is correct. This is the place where we can specify additional kernel arguments for bootc images. You can find more information in the upstream documentation.
• Various hygiene checks, such as non-UTF-8 filenames, checks on /etc and /usr/etc, unwanted logfiles in /var/log, and more.

Hence, if you work with bootc base images, make sure to run bootc container lint in the last command of your Containerfile.

GitHub Actions and disk space

Containers are commonly built in automated CI/CD pipelines. In another article, we elaborated on how to create such pipelines for image mode for RHEL to enable a GitOps-style user experience and workflow. If you are building images with GitHub Actions, you may run into disk-space related issues. Since bootc images ship an entire operating system, they tend to be larger than ordinary application images. Hence, if you run into such disk issues, make sure to add the following lines to your GitHub Actions workflow files:

# Based on https://github.com/orgs/community/discussions/25678
    - name: Delete huge unnecessary tools folder
        run: rm -rf /opt/hostedtoolcache

Removing the files in /opt/hostedtoolcache will free up a considerable amount of disk space that the bootc image can consume and ultimately help the container build to succeed.

Understanding /var

As outlined in the documentation, /var is really meant for persistent and mutable machine-local data and state. That means that during an update, /var will not be touched even when the container image has content in /var. Except for /var and /etc, all directories are mounted read-only, which is something we need to take into account when moving workloads over to image mode.

For instance, the httpd webserver wants to write data to /var/www at install time. The data is only meant to be read at run time and should be part of updates. Hence, we need to configure the container image accordingly by moving the directory to /usr/share/www. Here you can find an example Containerfile installing httpd and preparing the image.

Invoking useradd

Often, packaging scripts may invoke useradd. This can cause state drift when /etc/passwd is also locally modified on the system, and transient /etc is not in use. You can find more details on this state drift in the bootc documentation.

If a user does not own any content shipped in /usr and it runs as a systemd unit, then it’s often a good candidate to convert to systemd DynamicUser=yes, which has numerous advantages. Using DynamicUser will also help take care of ownership and more.

However, porting to DynamicUser=yes can be somewhat involved in complex cases. If the RPM contains files owned by the allocated user, but that content is only in /var/lib/somedaemon or /var/log/somedaemon; then often the best fix is to drop that content from the RPM (you can %ghost it to mark it as owned) and switch to creating it at run time via systemd-tmpfiles.

You can also switch to creating the user via systemd-sysusers. At that point, you can also drop the %post from the RPM which allocates the user.

When your package owns content shipped in /usr

Sometimes a daemon wants to drop privileges but also wants to access its configuration state in /etc. For example, polkit does this in /etc/polkit-1/rules.d. One solution is to use systemd’s BindReadOnlyPaths= option to mount the source directory into the namespace of the daemon.

If you run into the situation of depending on a setuid/setgid binary, then there is no solution other than statically allocating the user, which requires global coordination. If you are a package maintainer, you can officially request such static users as described in the Fedora docs.

Embedding containers with Quadlets

Running containerized workloads in systemd is a simple yet powerful means for reliable and rock-solid deployments. Podman has an excellent integration with systemd in the form of Quadlet. Quadlet is a tool for running Podman containers in systemd in an optimal and declarative way. Workloads can be declared in the form of systemd-unit-like files extended with Podman-specific functionality.

Quadlets integrate perfectly with image mode. You can find more details and examples in our previous article on containerizing workloads on image mode for RHEL. Note that using Quadlets at boot time may delay the boot process when new application container images need to be downloaded. However, there is a solution for that using so-called logically-bound images which will be pre-fetched during an update. Embedding containers via Quadlets works very well.

Summary

Using image mode is a paradigm shift in working with RHEL hosts. First, we can make use of all the great tools from the cloud-native world to build, deploy, and manage our operating system. Second, we are dealing with an immutable OS where large parts of the system are mounted read only. This article explained how to navigate that space.