Reduce bootc system update size

Sometimes it is necessary to minimize bandwidth usage when updating a system. Typically, when updating a bootc system, you download each modified image layer. This means a modification to a single configuration file could result in downloading gigabytes of data. This is where the rpm-ostree build-chunked-oci tool is helpful. It will automatically isolate related rpm packages into separate layers and output a new image. 

A new feature of the tool allows direct assigning of individual files to specific layers in the chunked image. Physically bound images (i.e., container images embedded in the bootc image) will benefit from this. By isolating the physically bound images into their own layer, modifications to unpackaged configuration files will not redownload the physically bound images during updates. This article will demonstrate how this works.

Set up a test environment

First, we need to set up a test environment. We will do this locally on our development machine, but this can easily be adapted to a production environment. Let’s start by building a new image with a large AI model embedded as a physically bound image. To add the physically bound image to a separate layer, all we need to do is set the user.component=containers xattr in the directory with the image.

Here is the Containerfile we will build:

FROM quay.io/fedora/fedora-bootc:42
COPY ./embed_image.sh /usr/bin/
RUN <<EOF
  set -euxo pipefail
  /usr/bin/embed_image.sh quay.io/ai-lab/mistral-7b-code-16k-qlora
  setfattr -n user.component -v "containers" /usr/lib/containers-image-cache
EOF

The embed_image.sh script is found in the fedora bootc examples repo. Save it in the same directory as the Containerfile as follows:

#!/bin/bash

set -euxo pipefail

image=$1
additional_copy_args=${2:-""}

mkdir -p /usr/lib/containers-image-cache
sha=$(echo "$image" | sha256sum | awk '{ print $1 }')
skopeo copy $additional_copy_args --preserve-digests docker://$image dir:/usr/lib/containers-image-cache/$sha
echo "$image,$sha" >> /usr/lib/containers-image-cache/mapping.txt

Now let's build and push the image to Quay:

podman build . -t quay.io/<user>/rechunker-test:latest
podman push quay.io/<user>/rechunker-test:latest

Finally, let’s run rpm-ostree build-chunked-oci on the image and push the chunked image to Quay.

rpm-ostree compose build-chunked-oci --bootc --format-version=2 --from=quay.io/<user>/rechunker-test:latest --output=containers-storage:quay.io/<user>/rechunker-test:chunked
podman push quay.io/<user>/rechunker-test:chunked

Update a system

Let’s see how a modification to /etc/hosts behaves when updating a system running the rechunker-test image. First, we will create a new VM using bcvk:

bcvk libvirt run --ssh --filesystem ext4 quay.io/<user>/rechunker-test:chunked

Now let’s modify the rechunker-test image by adding an sshd config file. Here is the updated Containerfile:

FROM quay.io/fedora/fedora-bootc:42
COPY ./embed_image.sh /usr/bin/
RUN <<EOF
set -euxo pipefail
/usr/bin/embed_image.sh quay.io/ai-lab/mistral-7b-code-16k-qlora
setfattr -n user.component -v "containers" /usr/lib/containers-image-cache
EOF

RUN <<EOF
set -euxo pipefail

cat > /etc/ssh/sshd_config.d/10-disable-password-auth.conf << END
Host *
  PasswordAuthentication no
END

EOF

We need to build, chunk, and push the image like before:

podman build . -t quay.io/<user>/rechunker-test:latest
podman push quay.io/<user>/rechunker-test:latest
rpm-ostree compose build-chunked-oci --bootc --format-version=2 --from=quay.io/<user>/rechunker-test:latest --output=containers-storage:quay.io/<user>/rechunker-test:chunked
podman push quay.io/<user>/rechunker-test:chunked

The image is updated in Quay so we can upgrade our test VM. Notice how it downloads a single layer a few megabytes in size. This layer is all of the unpackaged content except for the physically bound image we isolated to its own layer.

# run this on the test VM
sudo bootc upgrade

Now that we see how quick the update was, try running through the example without the xattr set in the Containerfile to see how long an update takes.

Final thoughts

Using the new rpm-ostree feature allows the assigning of individual files to specific layers in the output image. This is particularly beneficial for isolating physically bound images (i.e., large AI models) into their own layers, preventing them from being redownloaded when modifying other configuration files. Our demonstration showed the upgrade went much faster after making a simple change. Play around with this to see how best to optimize your specific images for faster updates.