When Red Hat Enterprise Linux (RHEL) is running in image mode and you want to make a change to its filesystem, you create a new image containing your changes. The system configurations that are different from the running image are merged to create a new default state of the operating system. A 3-way merge incorporates a third version, older than the current and new image, to minimize merge conflicts.
We typically talk about how filesystems are treated on disk differently in image mode than in package mode because it's key to how image mode operates. You’ve probably read or heard something like the list below:
/usr→ image state: Contents of the image will be extracted and overwrite local files/etc→ local configuration state: Contents of the image are merged with a preference for local files/var→ local state: Contents of the image will be ignored after the initial installation of any image
One of the common questions I hear when talking about image mode updates is “How exactly does image mode decide which updates from the image get applied and what stays the same in /etc during an upgrade?” To get to the answer, we’ll need to talk about some of the internal implementation details of bootc and OSTree, but by the end of this you should have a good basis for predicting how a 3-way merge will happen on a system, and maybe a few tricks to check for yourself.
What's in a merge anyway?
In image mode, we say that on image updates or switches, we merge the contents of the new image with the local files, with a preference for changes in local files. This means that if you set the hostname or IP address of the system, then a generic file in the new image won't reset all your systems to localhost on an upgrade. That makes sense. We set the hostname, after all, and bootc can detect that the two files don't match. It can detect which "direction" the change was made. But how does it know which one is right?
Because we said we prefer local files, we could just throw out anything in a new image, but that could cause other issues. For example, suppose there's a file in our new image that doesn't exist on the local machine. Is this a brand new config that should be added to the new /etc, or was the local file deliberately removed and therefore should be skipped?
The question of merge direction or correctness is not a new problem. The 2-way versus 3-way merge has been around since diff3 and CVS version control in the late 1980s. A 3-way merge makes automatic merging more reliable by adding, as the term suggests, a third version. Unlike CVS, Git, or other version systems, we aren't merging files on a line by line basis to create new composite files. We're looking to avoid merge conflicts that require human intervention, in favor of a predefined way of making these determinations. Comparisons then are simpler modified, added, or deleted files.
The key to making this work is that the third version must be older than either of the two versions we have on hand. So we have the local /etc and the new image's /etc, where do we get that third copy? For that, we turn to OSTree.
Where does /etc live?
You can think of OSTree as an engine for bootc, and it could be summarized as "Git for operating system binaries" (to quote the documentation). This is how we get multiple, parallel installations of independent operating systems on a single image mode host. The everyday operations are managed with bootc, which drives OSTree in the background. OSTree is where our rules for filesystem handling and this 3-way merge implementation comes from.
Each independent operating system on disk is called a deployment, and is essentially a complete filesystem that you could chroot into. Each one is a complete and bootable entity. If you look at the documentation for how OSTree creates a new deployment, you notice that it mentions a directory that's not a usual part of a Linux distribution: /usr/etc. The new inbound image will have an /etc directory, like normal, but no /usr/etc. You can check this by running ls in podman on a bootc image you have handy.
In OSTree, this directory is part of what’s known as the defaults or stateroot. On image mode hosts, this new stateroot is created by bootc during an update or switch to the new image.. Because the /usr/etc directory is a critical part of the merge, and doesn't exist in our bootc image, bootc maps /etc from the image to the /usr/etc directory in the new on-disk deployment. This is where we get our older copy that forms the third leg of this process.
On a running image mode host, you can look for local changes made to /etc using this diff command:
diff /etc /usr/etcIf it's available, you can alternately use the ostree command for the OSTree view of changes in the currently booted image:
ostree admin config-diffA word of warning: Using ostree directly requires root privileges and works below bootc. There are a few options you don't want to run manually or accidentally. Looking at the local state against the image default is fine, but do be careful with the command.
One important thing about bootc and deployments is their relationship to the bootloader. The bootloader gets pointed directly to the deployment directory, where all of the required artifacts are actually stored on disk. The filesystem you see when the image mode host is running is really a series of hard and soft links managed at boot. That means that the current local /etc on a running host is also hardlinked to the /etc in the corresponding deployment. When you issue a rollback, bootc tells the bootloader to boot the previous complete deployment, including its version of /etc. The 3-way merge we're talking about doesn't apply to rollbacks, only to upgrades or switching.
Bringing it all together
The players are assembled, so how does the scene run? Let's walk through it.
We run bootc upgrade on an image mode host to pull down a new image. Then bootc stages that image as follows:
- Creates a new OSTree deployment from the image contents
- Creates the new
/usr/etcdefaults for that deployment - Sets a flag to trigger the bootloader update and the
/etcmerge at next boot
This is important because we don’t want to merge changes on the live system, but they must be available at the next boot for the system to start. Remember that no changes are made to live hosts by bootc, only at reboots. We may also miss local changes made after an image was staged. When we check bootc status, we can see the new deployment listed, and this is what the staged status really means.
Once we reboot into the new deployment, the merge actually takes place as part of system startup. The 3-way merge combines the new defaults (/usr/etc of the deployment we are booting), the old defaults (/usr/etc of the previous deployment), and the local /etc (hardlinked /etc from the previous deployment) to create a new /etc on disk (Figure 1). There are no complicated state databases or other means in play, just direct comparison of copies of files on disk.
Exploring changes by hand
To get to our new desired /etc state, we first compare the two defaults, then we compare that against the local /etc. We can't really watch how the merge is actually done, but because these are all on disk after staging an update, we can use diff to compare the directories. On a bootc host with an update staged, but not applied, I ran the following sets of diffs. For readability, the staged image filepaths have the environment variable I set to make the command easier for me to type. You can find the location of any deployment on disk by looking for the OSTree checksum for an image in the full output of bootc status.
Let’s track some deliberate changes through our comparisons to predict what will happen at reboot. I added an authorized users banner to the message of the day (/etc/motd), deleted /etc/login.defs, and added a default rcfile for tcsh. Move, add, delete are the basic changes we’d be making to an image, but note that removing login.defs isn’t something you’d normally want to do. I'm just making sure we have a removed file.
Here's the section I added to the Containerfile to make these changes. It's not elegant, but it's functional enough to get the point across:
RUN rm /etc/login.defs
RUN echo "This is a private system. Unauthorized access is prohibited." > /etc/motd
COPY tchshrc /etc/skel/.tcshrcThe first comparison is between the defaults, using the newly staged image as the base. I’m using -q to only display that files differ and not the actual contents, and --no-dereference to skip any symlinks in the defaults that aren’t valid as a result of the copy process.
sudo diff -rq --no-dereference $STAGED_ROOT/usr/etc /usr/etc
Only in /usr/etc: hostname
** Only in /usr/etc: login.defs
** Files STAGED_ROOT/usr/etc/motd and /usr/etc/motd differ
Only in /usr/etc: resolv.conf
Files STAGED_ROOT/usr/etc/shadow and /usr/etc/shadow differ
Files STAGED_ROOT/usr/etc/shadow- and /usr/etc/shadow- differ
** Only in STAGED_ROOT/usr/etc/skel: .tcshrc
Files STAGED_ROOT/usr/etc/yum.repos.d/redhat.repo and /usr/etc/yum.repos.d/redhat.repo differFor the sake of this article, I’ve placed two asterisks before the changes made through the Containerfile. The other lines are just a result of the build process. Our Containerfile auto-generates a password for a break-glass user, which results in differences between /etc/shadow (because the seed changes).
Now, let's look at the current defaults and the local state, with the same diff options:
sudo diff -rq --no-dereference /usr/etc /etc
Only in /etc/containers: networks
Only in /etc/issue.d: 22_clhm_enp1s0.issue
Only in /etc: locale.conf
Files /usr/etc/machine-id and /etc/machine-id differ
Files /usr/etc/resolv.conf and /etc/resolv.conf differ
Only in /etc/rhsm/facts: bootc.facts
Only in /etc/ssh: ssh_host_ecdsa_key
Only in /etc/ssh: ssh_host_ecdsa_key.pub
Only in /etc/ssh: ssh_host_ed25519_key
Only in /etc/ssh: ssh_host_ed25519_key.pub
Only in /etc/ssh: ssh_host_rsa_key
Only in /etc/ssh: ssh_host_rsa_key.pub
Only in /etc/tmpfiles.d: bootc-root-ssh.conf
Only in /etc: vconsole.confMost of these changes are install-time or runtime changes. The files that we're tracking don't show up in this comparison, which means they are the same in the current defaults and in the local state.
What about between the new defaults and the local state?
sudo diff -rq --no-dereference $STAGED_ROOT/usr/etc /etc
Only in /etc/containers: networks
Only in /etc: hostname
Only in /etc/issue.d: 22_clhm_enp1s0.issue
Only in /etc: locale.conf
** Only in /etc: login.defs
Files STAGED_ROOT/usr/etc/machine-id and /etc/machine-id differ
** Files STAGED_ROOT/usr/etc/motd and /etc/motd differ
Only in /etc: resolv.conf
Only in /etc/rhsm/facts: bootc.facts
Files STAGED_ROOT/usr/etc/shadow and /etc/shadow differ
Files STAGED_ROOT/usr/etc/shadow- and /etc/shadow- differ
** Only in STAGED_ROOT/usr/etc/skel: .tcshrc
Only in /etc/ssh: ssh_host_ecdsa_key
Only in /etc/ssh: ssh_host_ecdsa_key.pub
Only in /etc/ssh: ssh_host_ed25519_key
Only in /etc/ssh: ssh_host_ed25519_key.pub
Only in /etc/ssh: ssh_host_rsa_key
Only in /etc/ssh: ssh_host_rsa_key.pub
Only in /etc/tmpfiles.d: bootc-root-ssh.conf
Only in /etc: vconsole.conf
Files STAGED_ROOT/usr/etc/yum.repos.d/redhat.repo and /etc/yum.repos.d/redhat.repo differOur new tcshrc shows up as just in the new defaults, we can see there’s a difference between the MOTD files, and that login.defs is only in the new defaults. In a simple (or 2-way) merge, we'd be unsure what updates to make. Should we change the MOTD? Is deleting login.defs the right thing to do? What about /etc/shadow or the machine-id? We know what those changes should be, but we need to make sure this is automated and doesn't require human intervention at boot.
By adding the current defaults (the state created at build time for the image) to the comparison, we can make our policy based decisions with more predictable outcomes.
Combinations and outcome
Generally, files in the new defaults win, unless there was a local change in /etc. Changes to /etc get carried along for the life of the host, even if the software installed is radically different as the result of a bootc switch. If there's a file removed in the new defaults, and no local changes are made, that file is removed.
New defaults | Exiting defaults | Local state | Result |
ANY | Exists | Changed | Local kept |
Modified | Exists | Exists | New kept |
Added | Missing | Missing | Added |
Deleted | Exists | Exists | Deleted |
Caveats and party tricks
Now I need to reiterate that we've dug into internal implementation details here, and not something you should be poking at regularly. Don't add something to a Containerfile in /usr/etc (to enforce a new default, for instance), or you’ll end up with problems, because this results in a literally undefined behavior. This is an excellent reason to add RUN bootc container lint to the end of every image mode Containerfile. That warns you about potential problems like this one.
Resetting a file to image control
However, you could put a file back under control of image updates and eliminate local modifications by copying a file from the defaults into /etc. Again, this is not something you'll need regularly, but a good thing to have in your toolbox.
For example, what if you wanted to start controlling admin privileges with a drop-in file in /etc/sudoers.d centrally managed in the image, but had been editing /etc/sudoers locally? An update would have both sets of privileges, the new drop-in file would be created because it wasn’t in the defaults, but the local changes wouldn’t be removed. This risks configuration drift. But because the /etc/sudoers that came with the RPM is in the image defaults, you could copy that over the local modifications and return /etc/sudoers to image control.
Changing baselines on deployed hosts
This can also be used to distribute a simple image controlled configuration change across multiple existing hosts without the need to roll a new image and schedule reboots.
For example, if the NTP pool needed to be updated fleet wide and in the standard build, you could create and test the new /etc/chrony.conf in the image build. Once that works for new deployments, you can copy that exact file from the build repository to the existing systems. Be sure it's identical, using something to create it locally might have extra headers not present in the image. For the first merge, this local copy of /etc/chrony.conf would be considered a local modification and kept. However, on the next update, since these two files are now identical according to the updated defaults, control over that file is returned to the image.
3-way merge demystified
In my experience, there’s only a handful of times I’ve needed to think about more than just incoming /etc and local /etc when trying to understand changes that were made (or not made, for that matter). I hope this has been helpful for getting a deeper understanding of just how a 3-way merge happens in the image mode context. This can be crucial when designing your standard images and deciding what to manage with the image, and what to manage on a host by host basis. This also applies when integrating image mode operations into existing automation and configuration management. Being able to split cleanly between build time and run time management helps smooth that integration work.
