In part one of this series, I introduced Fedora CoreOS (and Red Hat CoreOS) and explained why its immutable and atomic nature is important for running containers. I then walked you through getting Fedora CoreOS, creating an Ignition file, booting Fedora CoreOS, logging in, and running a test container. In this article, I will walk you through customizing Fedora CoreOS and making use of its immutable and atomic nature.
Extending Ignition files
In part one, we saw a basic example with a minimal Ignition file that we generated from an FCC file and then injected a public SSH key. We extend this example by adding more logic; for example, the creation of a systemd unit. Since we are working on a container-optimized system why not create a systemd unit with Podman? Thanks to the daemonless nature of Podman we can run, start or stop our containers in a systemd unit and manage their startup order easily.
For this purpose, I extended the previous FCC file:
variant: fcos
version: 1.0.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB3Nza...
systemd:
units:
- name: hello.service
enabled: true
contents: |
[Unit]
Description=A hello world unit!
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
KillMode=none
Restart=on-failure
RemainAfterExit=yes
ExecStartPre=podman pull quay.io/gbsalinetti/hello-server
ExecStart=podman run -d --name hello-server -p 8080:8080 quay.io/gbsalinetti/hello-server
ExecStop=podman stop -t 10 hello-server
ExecStopPost=podman rm hello-server
[Install]
WantedBy=multi-user.target
This time, I added a simple unit file to launch a self-made image that runs a minimal and ephemeral Go web server and then prints a "Hello World" message. The source code for the example can be found here. Take a look at the file's systemd section and its units subsection, which contains a list of items representing one or more unit files. In this block, we can define as many units as we need, and they will be created and started at boot.
We can use Ignition configs to manage storage and users; create files, directories, and systemd units; and inject ssh keys. Detailed syntax documentation with examples of Ignition syntax can be found in the fcct-config documentation.
After customizing the FCC file, we must generate the Ignition file using the fcct tool:
$ fcct -input example-fcc-systemd.yaml -output example-ignition-systemd.json
Testing the new instance
We are ready to apply the generated Ignition file to the FCOS new instance using the virt-install command:
$ sudo virt-install --connect qemu:///system \ -n fcos -r 2048 --os-variant=fedora31 --import \ --graphics=none \ --disk size=10,backing_store=/home/gbsalinetti/Labs/fedora-coreos/fedora-coreos-31.20200118.3.0-qemu.x86_64.qcow2 \ --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/home/gbsalinetti/Labs/fedora-coreos/example-ignition-systemd.ign"
At the end of the boot process, let’s log in and check if the container is running (update your IP address accordingly in the ssh command):
$ ssh core@192.168.122.11 Fedora CoreOS 31.20200118.3.0 Tracker: https://github.com/coreos/fedora-coreos-tracker Last login: Fri Feb 7 23:22:31 2020 from 192.168.122.1 [core@localhost ~]$ systemctl status hello.service ● hello.service - A hello world unit! Loaded: loaded (/etc/systemd/system/hello.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2020-02-07 23:18:39 UTC; 12min ago Process: 2055 ExecStartPre=/usr/bin/podman pull quay.io/gbsalinetti/hello-server (code=exited, status=0/SUCCESS) Process: 2112 ExecStart=/usr/bin/podman run -d --name hello-server -p 8080:8080 quay.io/gbsalinetti/hello-server (code=exited, status=0/SUCCESS) Main PID: 2112 (code=exited, status=0/SUCCESS) Feb 07 23:18:17 localhost podman[2055]: Writing manifest to image destination Feb 07 23:18:17 localhost podman[2055]: Storing signatures Feb 07 23:18:38 localhost podman[2055]: 2020-02-07 23:18:38.671593577 +0000 UTC m=+47.966065770 image pull Feb 07 23:18:38 localhost podman[2055]: 146c93bfc4df81797068fdc26ee396348ba8c83a2d21b2d7dffc242dcdf38adb Feb 07 23:18:38 localhost systemd[1]: Started A hello world unit!. Feb 07 23:18:39 localhost podman[2112]: 2020-02-07 23:18:39.020399261 +0000 UTC m=+0.271239416 container create 2abf8d30360c03aead01092bbd8a8a51182a603911aac> Feb 07 23:18:39 localhost podman[2112]: 2020-02-07 23:18:39.801631894 +0000 UTC m=+1.052472079 container init 2abf8d30360c03aead01092bbd8a8a51182a603911aac9f> Feb 07 23:18:39 localhost podman[2112]: 2020-02-07 23:18:39.845449198 +0000 UTC m=+1.096289478 container start 2abf8d30360c03aead01092bbd8a8a51182a603911aac9> Feb 07 23:18:39 localhost podman[2112]: 2abf8d30360c03aead01092bbd8a8a51182a603911aac9f8b4f5a465f0360f05 Feb 07 23:18:39 localhost systemd[1]: hello.service: Succeeded.
We have launched a container as a one-shot Podman command. In the example, the command exited with a 0/SUCCESS status and the container was started successfully.
To inspect the running container, we can use the podman ps command:
[core@localhost ~]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2abf8d30360c quay.io/gbsalinetti/hello-server:latest hello-server 14 minutes ago Up 14 minutes ago 0.0.0.0:8080->8080/tcp hello-server
Why use sudo? We launched Podman under systemd, so the command was executed with root user privileges. This result is not a rootless container, also called rootful container. (A rootless container is executed with standard user privileges and a rootful container is executed with root privileges.)
The rpm-ostree tool
Now that we have played with Ignition files, let us learn how to handle system changes starting with RPM packages installation, and learn how package installs are handled in FCOS. If we try to launch the yum command in Fedora CoreOS we receive a "command not found" error. We get this same result running dnf. Instead, the utility managing packages in this kind of architecture must wrap RPM-based package management on top of an atomic file system management library. Every package change must be committed to the file system.
We already mentioned the rpm-ostree tool before. Let’s use it to apply a persistent change to the base OS image and then install the buildah package. Buildah is a great tool for managing Open Container Initiative (OCI) image builds because it replicates all of the instructions found in a Dockerfile, allowing us to build images with or without a Dockerfile while not requiring root privileges.
Installing new packages
A short recap before installing: There is no dnf or yum tool installed in FCOS, and rpm-ostree (built on top of the libostree library) is the default package manager. Changes are committed internally and systems are rebooted to apply the new layer.
Let’s check the status of the system prior to installation:
[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
Version: 31.20200118.3.0 (2020-01-28T16:10:53Z)
Commit: 093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
We can see only one layer here, with a specific commit ID. The rpm-ostree status --json command can be used to print extended statuses.
Now, let’s install the buildah package with the rpm-ostree install command:
$ sudo rpm-ostree install buildah
Let’s check the rpm-ostree status output again:
[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
ostree://fedora:fedora/x86_64/coreos/stable
Version: 31.20200118.3.0 (2020-01-28T16:10:53Z)
BaseCommit: 093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Diff: 1 added
LayeredPackages: buildah
● ostree://fedora:fedora/x86_64/coreos/stable
Version: 31.20200118.3.0 (2020-01-28T16:10:53Z)
Commit: 093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
The new commit added a new layer and the list of layered packages shows the Buildah package we installed before. We can then layer as many packages as we want; for example, tcpdump:
[core@localhost ~]$ sudo rpm-ostree install tcpdump
When we run the rpm-ostree status command again:
[core@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
ostree://fedora:fedora/x86_64/coreos/stable
Version: 31.20200118.3.0 (2020-01-28T16:10:53Z)
BaseCommit: 093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Diff: 2 added
LayeredPackages: buildah tcpdump
● ostree://fedora:fedora/x86_64/coreos/stable
Version: 31.20200118.3.0 (2020-01-28T16:10:53Z)
Commit: 093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e
GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
The new commit has two layered packages. Let’s launch buildah now:
[core@localhost ~]$ buildah --help -bash: buildah: command not found
Ouch, it looks like the binary isn’t installed yet. Once again, this response is correct because the system is still running on top of the base layer. To solve this problem and see the layered packages, we need to restart the system so it boots into the new file system layer:
[core@localhost ~]$ sudo systemctl reboot
Now we can enjoy the layered packages we installed before as they appear in the file system after the reboot.
How package management works
When installing packages, we are not really creating a new commit. Instead, we are layering the packages on top of the current system commit with a new deployment. Those layered packages will persist across updates and rebases. In other words, rpm-ostree fuses the best of image layers and package management.
Rolling back changes
The great thing about an atomic system the support for atomic rollbacks. This feature is useful when we reach an unstable configuration after an update and need to go back immediately to a working and stable system:
[core@localhost ~]$ sudo rpm-ostree rollback Moving '093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e.0' to be first deployment Transaction complete; bootconfig swap: no; deployment count change: 0 Removed: buildah-1.12.0-2.fc31.x86_64 tcpdump-14:4.9.3-1.fc31.x86_64 Run "systemctl reboot" to start a reboot
Once again, a reboot is necessary to boot the system with the rolled back layer.
Uninstalling
If we need to uninstall a single package without rolling everything back, we can use the rpm-ostree uninstall command:
[core@localhost ~]$ sudo rpm-ostree uninstall buildah Staging deployment... done Removed: buildah-1.12.0-2.fc31.x86_64 Run "systemctl reboot" to start a reboot
After reboot, the package won't be available anymore.
More in-depth with OSTree
The libostree library offers an API to manipulate atomic transactions on
immutable file systems by managing them as whole trees. The ostree command is the default tool used to manage these changes. There are many language bindings useful to create our own implementations; for example, ostree-go is the language binding for Golang.
Listing branches
We can see the delta between the two file system layers with the ostree refs command. First, we need to locate the different refs to the committed layers:
[core@localhost ~]$ ostree refs rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 rpmostree/base/0 ostree/0/1/1 rpmostree/pkg/tcpdump/14_3A4.9.3-1.fc31.x86__64 fedora:fedora/x86_64/coreos/stable ostree/0/1/0
Think of the refs as different branches, like in a Git repository. Notice the branches created by rpm-ostree, beginning with the pattern rpmostree/pkg/.
Inspecting refs content
To inspect the changes that happened in a branch, we can use the ostree ls command. For example, to see the files and directory changed by the buildah package:
[core@localhost ~]$ ostree ls -R rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 d00755 0 0 0 / d00755 0 0 0 /usr d00755 0 0 0 /usr/bin -00755 0 0 37474136 /usr/bin/buildah d00755 0 0 0 /usr/lib d00755 0 0 0 /usr/lib/.build-id d00755 0 0 0 /usr/lib/.build-id/db l00777 0 0 0 /usr/lib/.build-id/db/7c450ca346e503747a530a7185d9c16dc9a132 -> ../../../../usr/bin/buildah d00755 0 0 0 /usr/share d00755 0 0 0 /usr/share/bash-completion d00755 0 0 0 /usr/share/bash-completion/completions -00644 0 0 22816 /usr/share/bash-completion/completions/buildah d00755 0 0 0 /usr/share/doc d00755 0 0 0 /usr/share/doc/buildah -00644 0 0 8450 /usr/share/doc/buildah/README.md d00755 0 0 0 /usr/share/licenses d00755 0 0 0 /usr/share/licenses/buildah -00644 0 0 11357 /usr/share/licenses/buildah/LICENSE d00755 0 0 0 /usr/share/man d00755 0 0 0 /usr/share/man/man1 -00644 0 0 719 /usr/share/man/man1/buildah-add.1.gz -00644 0 0 10290 /usr/share/man/man1/buildah-bud.1.gz -00644 0 0 2264 /usr/share/man/man1/buildah-commit.1.gz -00644 0 0 2426 /usr/share/man/man1/buildah-config.1.gz -00644 0 0 1359 /usr/share/man/man1/buildah-containers.1.gz -00644 0 0 662 /usr/share/man/man1/buildah-copy.1.gz -00644 0 0 8654 /usr/share/man/man1/buildah-from.1.gz -00644 0 0 1691 /usr/share/man/man1/buildah-images.1.gz -00644 0 0 891 /usr/share/man/man1/buildah-info.1.gz -00644 0 0 635 /usr/share/man/man1/buildah-inspect.1.gz -00644 0 0 1029 /usr/share/man/man1/buildah-login.1.gz -00644 0 0 637 /usr/share/man/man1/buildah-logout.1.gz -00644 0 0 1051 /usr/share/man/man1/buildah-manifest-add.1.gz -00644 0 0 856 /usr/share/man/man1/buildah-manifest-annotate.1.gz -00644 0 0 615 /usr/share/man/man1/buildah-manifest-create.1.gz -00644 0 0 334 /usr/share/man/man1/buildah-manifest-inspect.1.gz -00644 0 0 907 /usr/share/man/man1/buildah-manifest-push.1.gz -00644 0 0 459 /usr/share/man/man1/buildah-manifest-remove.1.gz -00644 0 0 608 /usr/share/man/man1/buildah-manifest.1.gz -00644 0 0 1072 /usr/share/man/man1/buildah-mount.1.gz -00644 0 0 2144 /usr/share/man/man1/buildah-pull.1.gz -00644 0 0 2440 /usr/share/man/man1/buildah-push.1.gz -00644 0 0 200 /usr/share/man/man1/buildah-rename.1.gz -00644 0 0 363 /usr/share/man/man1/buildah-rm.1.gz -00644 0 0 905 /usr/share/man/man1/buildah-rmi.1.gz -00644 0 0 4268 /usr/share/man/man1/buildah-run.1.gz -00644 0 0 225 /usr/share/man/man1/buildah-tag.1.gz -00644 0 0 298 /usr/share/man/man1/buildah-umount.1.gz -00644 0 0 1111 /usr/share/man/man1/buildah-unshare.1.gz -00644 0 0 312 /usr/share/man/man1/buildah-version.1.gz -00644 0 0 2539 /usr/share/man/man1/buildah.1.gz
Single directories from a branch can be listed as well:
[core@localhost ~]$ ostree ls -R rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 /usr/bin d00755 0 0 0 /usr/bin -00755 0 0 37830616 /usr/bin/buildah
We can also check out a branch to an external directory with the ostree checkout command. The following example applies to the buildah ref:
[core@localhost ~]$ sudo ostree checkout rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 /tmp/buildah_checkout
After running the above command the /tmp/buildah_checkout folder will contain all the files checked out from the rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 branch. This is a great feature for debugging and troubleshooting.
Rebasing
Now that we have introduced the ostree command, we can show an amazing feature: rebasing. With rebasing, we can move to a different branch and radically modify our file system tree.
Ever wonder how to switch from a stable release to a testing release and then to a rawhide without reinstalling? With rebasing it's possible!
The rpm-ostree rebase command can take a branch as an argument and move the whole system to that branch. If we recall the output of the ostree refs command, there was the following line: fedora:fedora/x86_64/coreos/stable. This output means that we are on the stable branch of Fedora CoreOS. If we want to switch to the testing branch, we just have to rebase, like in the following example.
[core@localhost ~]$ sudo rpm-ostree rebase -b fedora:fedora/x86_64/coreos/testing ⠒ Receiving objects: 99% (5030/5031) 627.5 kB/s 151.2 MB Receiving objects: 99% (5030/5031) 627.5 kB/s 151.2 MB... done Staging deployment... done Upgraded: NetworkManager 1:1.20.8-1.fc31 -> 1:1.20.10-1.fc31 NetworkManager-libnm 1:1.20.8-1.fc31 -> 1:1.20.10-1.fc31 e2fsprogs 1.45.3-1.fc31 -> 1.45.5-1.fc31 e2fsprogs-libs 1.45.3-1.fc31 -> 1.45.5-1.fc31 fuse-overlayfs 0.7.3-2.fc31 -> 0.7.5-2.fc31 glibc 2.30-8.fc31 -> 2.30-10.fc31 glibc-all-langpacks 2.30-8.fc31 -> 2.30-10.fc31 glibc-common 2.30-8.fc31 -> 2.30-10.fc31 kernel 5.4.10-200.fc31 -> 5.4.13-201.fc31 kernel-core 5.4.10-200.fc31 -> 5.4.13-201.fc31 kernel-modules 5.4.10-200.fc31 -> 5.4.13-201.fc31 libcom_err 1.45.3-1.fc31 -> 1.45.5-1.fc31 libsolv 0.7.10-1.fc31 -> 0.7.11-1.fc31 libss 1.45.3-1.fc31 -> 1.45.5-1.fc31 pcre2 10.33-16.fc31 -> 10.34-4.fc31 selinux-policy 3.14.4-43.fc31 -> 3.14.4-44.fc31 selinux-policy-targeted 3.14.4-43.fc31 -> 3.14.4-44.fc31 socat 1.7.3.3-2.fc31 -> 1.7.3.4-1.fc31 toolbox 0.0.17-1.fc31 -> 0.0.18-1.fc31 whois-nls 5.5.4-1.fc31 -> 5.5.4-2.fc31 zchunk-libs 1.1.4-1.fc31 -> 1.1.5-1.fc31 Removed: buildah-1.12.0-2.fc31.x86_64 Run "systemctl reboot" to start a reboot
Notice the list of packages upgraded to switch from the stable to the testing branch. Now, the ostree refs output is slightly different, with the testing branch replacing stable:
[core@localhost ~]$ ostree refs rpmostree/pkg/buildah/1.12.0-2.fc31.x86__64 rpmostree/base/0 ostree/0/1/1 rpmostree/pkg/tcpdump/14_3A4.9.3-1.fc31.x86__64 fedora:fedora/x86_64/coreos/testing ostree/0/1/0
This release streams link contains a more detailed discussion of how branches are managed.
File system analysis
It is easy to notice that some directories are not writable; for example, when trying to write to /usrwe get a read-only file system error. When the system boots, only certain portions (like /var) are made writable. So, how does ostree manage files?
The ostree architecture design states that the system read-only content is kept in the /usr directory. The /var directory is shared across all system deployments and is writable by processes, and there is an/etc directory for every system deployment. When system changes or upgrades, previous modifications in the/etc directory are merged with the copy in the new deployment.
All of these changes are stored in an OSTree repository in /ostree/repo, which is a symlink to /sysroot/ostree/repo. This directory stores all system deployments. Think of /sysroot/ostree/repo like the .gitdirectory in a Git repository.
To check the current status and identify the active deployment UUID, use the following command:
[core@localhost ~]$ sudo ostree admin status
* fedora-coreos 4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505.0
Version: 31.20200210.3.0
origin refspec: fedora:fedora/x86_64/coreos/stable
If we inspect the /ostree folder in a freshly installed system, we find an exact match that contains our booted file system tree:
[core@localhost ~]$ ls -al /ostree/boot.1/fedora-coreos/19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/0/ total 8 drwxr-xr-x. 12 root root 253 Feb 24 16:52 . drwxrwxr-x. 3 root root 161 Feb 24 16:52 .. lrwxrwxrwx. 2 root root 7 Feb 24 16:51 bin -> usr/bin drwxr-xr-x. 2 root root 6 Jan 1 1970 boot drwxr-xr-x. 2 root root 6 Jan 1 1970 dev drwxr-xr-x. 77 root root 4096 Mar 11 08:54 etc lrwxrwxrwx. 2 root root 8 Feb 24 16:51 home -> var/home lrwxrwxrwx. 2 root root 7 Feb 24 16:51 lib -> usr/lib lrwxrwxrwx. 2 root root 9 Feb 24 16:51 lib64 -> usr/lib64 lrwxrwxrwx. 2 root root 9 Feb 24 16:51 media -> run/media lrwxrwxrwx. 2 root root 7 Feb 24 16:51 mnt -> var/mnt lrwxrwxrwx. 2 root root 7 Feb 24 16:51 opt -> var/opt lrwxrwxrwx. 2 root root 14 Feb 24 16:51 ostree -> sysroot/ostree drwxr-xr-x. 2 root root 6 Jan 1 1970 proc lrwxrwxrwx. 2 root root 12 Feb 24 16:51 root -> var/roothome drwxr-xr-x. 2 root root 6 Jan 1 1970 run lrwxrwxrwx. 2 root root 8 Feb 24 16:51 sbin -> usr/sbin lrwxrwxrwx. 2 root root 7 Feb 24 16:51 srv -> var/srv drwxr-xr-x. 2 root root 6 Jan 1 1970 sys drwxr-xr-x. 2 root root 6 Jan 1 1970 sysroot drwxrwxrwt. 2 root root 6 Mar 11 08:37 tmp drwxr-xr-x. 12 root root 155 Jan 1 1970 usr drwxr-xr-x. 4 root root 28 Mar 11 08:37 var
The above directory represents the booted deployment and is actually a symbolic link to a ref under /ostree/deploy/fedora-cores/deploy/. Notice that the directory's name will match with the UUID printed by the ostree admin status command's output.
At system boot, some directories in the filesystem root are created as hard links to the active deployment tree. Since hard links point to the same inode number in the file system, let's cross-check the inode number of the /usr folder and the one in the booted deployment:
[core@localhost ~]$ ls -alid /usr 3218784 drwxr-xr-x. 12 root root 155 Jan 1 1970 /usr [core@localhost ~]$ ls -alid /sysroot/ostree/boot.1/fedora-coreos/19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/0/usr/ 3218784 drwxr-xr-x. 12 root root 155 Jan 1 1970 /sysroot/ostree/boot.1/fedora-coreos/19190477fad0e60d605a623b86e06bb92aa318b6b79f78696b06f68f262ad5d6/0/usr/
As expected, the inode number 3218784 is the same for both directories, demonstrating that the content under the filesystem root is composed of the active deployment.
What happened when we installed the Buildah package with rpm-ostree? After rebooting, a new deployment will appear:
[core@localhost ~]$ sudo ostree admin status * fedora-coreos ee678bde3c15d8cae34515e84e2b4432ba3d8c9619ca92c319b576a13029481d.0 Version: 31.20200210.3.0 origin: <unknown origin type> fedora-coreos 4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505.0 (rollback) Version: 31.20200210.3.0 origin refspec: fedora:fedora/x86_64/coreos/stable
Notice the star next to the current active deployment. We expect to find it under /ostree/deploy/fedora-cores/deployalong with the old one:
[core@localhost ~]$ ls -al /ostree/deploy/fedora-coreos/deploy/ total 12 drwxrwxr-x. 4 root root 4096 Mar 11 10:18 . drwxrwxr-x. 4 root root 31 Feb 24 16:52 .. drwxr-xr-x. 12 root root 253 Feb 24 16:52 4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505.0 -rw-r--r--. 1 root root 52 Feb 24 16:52 4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505.0.origin drwxr-xr-x. 12 root root 253 Mar 11 10:01 ee678bde3c15d8cae34515e84e2b4432ba3d8c9619ca92c319b576a13029481d.0 -rw-r--r--. 1 root root 87 Mar 11 10:18 ee678bde3c15d8cae34515e84e2b4432ba3d8c9619ca92c319b576a13029481d.0.origin
The /ostree/deploy/fedora-cores directory is also called stateroot or osname, and it contains the deployment and all its related commits. In every stateroot directory there is only one var directory that is mounted under /var using a systemd mount unit (repesented by the file/run/systemd/generator/var.mount).
Let's finish this file system deep dive with a look into the /ostree/repo folder. This is where we can find the closest similarities with Git:
[core@localhost repo]$ ls -al /ostree/repo/ total 16 drwxrwxr-x. 7 root root 102 Mar 11 10:18 . drwxrwxr-x. 5 root root 62 Mar 11 10:18 .. -rw-r--r--. 1 root root 73 Feb 24 16:52 config drwxr-xr-x. 3 root root 23 Mar 11 09:59 extensions -rw-------. 1 root root 0 Feb 24 16:51 .lock drwxr-xr-x. 258 root root 8192 Feb 24 16:52 objects drwxr-xr-x. 5 root root 49 Feb 24 16:51 refs drwxr-xr-x. 2 root root 6 Feb 24 16:52 state drwxr-xr-x. 3 root root 19 Mar 11 10:18 tmp
Notice the refs and objects folders, which store respectively the branch information and the objects versioned. Here, we have an exact matching within the .git folder of a typical Git repository.
Managing system upgrades
It should be clear by now that rpm-ostree works on top of the ostree library and provides packages management with an atomic approach. System upgrades are also atomic. Upgrading a system is just layering a new commit on top of the existing file system, and we can do it easily with the rpm-ostree upgrade command:
[core@localhost ~]$ sudo rpm-ostree upgrade -r
Note: The -r flag tells rpm-ostree to automatically reboot after the upgrade is complete.
Despite letting users upgrade systems manually, FCOS provides a dedicated service that manages system upgrades called Zincati. Zincati is an agent that performs periodic upgrade checks and applies them. We can check our Zincati service status with the systemctl command:
[core@localhost ~]$ sudo systemctl status zincati
● zincati.service - Zincati Update Agent
Loaded: loaded (/usr/lib/systemd/system/zincati.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-02-07 11:36:33 UTC; 3min 30s ago
Docs: https://github.com/coreos/zincati
Main PID: 707 (zincati)
Tasks: 2 (limit: 2297)
Memory: 17.5M
CGroup: /system.slice/zincati.service
└─707 /usr/libexec/zincati agent -v
Feb 07 11:36:33 localhost systemd[1]: Started Zincati Update Agent.
Feb 07 11:36:33 localhost zincati[707]: [INFO ] starting update agent (zincati 0.0.6)
Feb 07 11:36:39 localhost zincati[707]: [INFO ] Cincinnati service: https://updates.coreos.stg.fedoraproject.org
Feb 07 11:36:39 localhost zincati[707]: [INFO ] agent running on node '20d1f6332922438d8a8edede3fbe6251', in update group 'default'
Feb 07 11:36:39 localhost zincati[707]: [INFO ] initialization complete, auto-updates logic enabled
Zincati behavior can be customized. Default configs are already installed under /usr/lib/zincati/config.d/, while users can apply custom configs in /etc/zincati/configs.d/ and override the defaults.
OpenShift 4 and the Machine Config Operator
The developer community is working toward integrating OKD 4 and Fedora CoreOS and we are waiting the first stable release of OKD 4 very soon.
Nowadays, all Red Hat OpenShift Container Platform 4 (RHOCP), run on top of Red Hat CoreOS (RHCOS) nodes. It’s useful to understand how RHCOS systems are managed. Let’s start with a provocation. In RHOCP 4, no sysadmin should SSH to the RHCOS or Fedora CoreOS nodes to make changes.
Who takes care of node management, upgrades, and who applies Ignition configs? All of these tasks are managed in OpenShift by the Machine Config Operator (MCO), which was already mentioned in the previous article.
The MCO is a core opeator and it spawns different components in the OpenShift cluster:
machine-config-server(MCS) serves the Ignition files to the nodes via HTTPS.machine-config-controller(MCC) coordinates upgrades of machines to their desired configurations as defined byMachineConfigobjects.machine-config-daemon(MCD) runs on every node as a DaemonSet, applying machine configurations and validating the machines' state to the requested configurations.
MCD performs configurations defined in the provided Ignition files using CoreOS technology. For more details about the machine-config-daemon, read this documentation.
The MCD manages system upgrades using Podman to pull and mount system images and rpm-ostree rebase to rebase RHCOS nodes to the mounted container's file system trees. In other words, OCI images are used to transport whole, upgraded file systems to nodes.
The MachineConfig objects processed by the MCD are nothing but OpenShift resources that embed Ignition configs. MachineConfigs are assigned to a MachineConfigPools and applied on all the machines belonging to that pool. Pools have a direct match with node roles in the cluster and by default we have only 2 MachineConfigPools in OCP 4, masterand worker, but it is possible to add custom pools that reflect specific roles, for example infra nodes or HPC nodes.
The Machine Config Operator and the components it manages were created with the immutable and atomic approach in mind, implemented on top of Red Hat CoreOS in order to automate the day2 operations on nodes and deliver a NoOps container platform to the customer.
The MCO architecture brings great value in hybrid cloud environment where the infrastructure automation is almost mandatory to manage complex scenarios.
Conclusions
This journey into the features of Fedora CoreOS and immutable systems has ended, but these two articles are just the beginning. Immutable infrastructures are the next big thing and not only in containerized workloads. I think that with the correct tooling they could be game-changers in traditional, bare metal, and on-prem scenarios.
By having reliable systems that are rebuilt rather than changed and managing images with a Git-like approach with commits, branches, and rollbacks, we can truly embrace the culture of Infrastructure-as-Code. This approach opens us to systems that are more maintainable, sustainable, and stable.
Last updated: March 29, 2023