Self-encrypting VM images for confidential environments hands on

Confidential VMs and storage encryption

September 24, 2026
Vitaly Kuznetsov
Related topics:
ContainersSecurity
Related products:
Image mode for Red Hat Enterprise Linux

Confidential virtual machine (VM) technologies like AMD's SEV-SNP and Intel's TDX are becoming increasingly popular in various computing environments, such as public and private clouds. These technologies are designed for data protection in runtime (for example, to ensure that a potentially malicious virtualization host cannot steal confidential data from a VM's memory or CPU registers). However, protecting persistent storage is, as they say, "left as an exercise to the reader" (the guest operating system, in this context). Linux has provided mechanisms for verity protection (dm-verity, fs-verity) and disk encryption (dm-crypt, LUKS) for years, but applying these mechanisms correctly in a confidential environment can be challenging.

Consider a situation where a single golden image of a traditional (mutable) operating system is used to create multiple confidential VM instances. The basic requirements for the VM storage would be:

  • The fact that the genuine base image was used for creating a VM must be verifiable/attestable.
  • All mutable parts of the OS must be encrypted before their first use. Ideally, this needs to be complemented with integrity and replay attack protection.
  • No 2 VM instances may share the same encryption key. A compromise of any single VM does not affect another VM.

A simple scheme to achieve these goals:

  • The base image is verity-protected with dm-verity: The content remains visible and accessible to the host, but a Merkle tree provides runtime integrity of the data.
  • On first boot, an encrypted empty overlay is created and mounted over the VM's root.
  • On subsequent boots, the previously created overlay is used.

In this article, we demonstrate how to implement this using Fedora Rawhide.

Design

We're going to create a VM image with this partition layout (see figure 1 for a visualization):

  • EFI system partition
  • /usr
  • dm-verity for /usr
  • Empty space

Figure 1: The initial VM image partition layout makes space for EFI, /usr, and dm-verity.

On the first boot, systemd-repart in the VM's initramfs uses the empty space to create a new dm-crypt encrypted root (/) partition:

Figure 2: The VM image partition layout, after the first boot, contains a dm-crypt partition for the root.

Our goal is to make the whole image writable, including the /usr partition, so we have 2 choices:

  • Do a full copy of the dm-verity content to the new root partition.
  • Create a writable overlay for /usr.

Performing a full copy is simple, but not very practical from a boot-time perspective. Public clouds normally offer network-attached storage for the VM root partition, and a full copy may easily take minutes to complete. The overlay solution used in this article keeps the base image immutable for the lifetime of the VM, with all modifications to the /usr filesystem persisted on the encrypted partition created at first boot. Overlayfs takes care of assembling the resulting filesystem, which is available to all applications in read-write mode. This approach allows us to significantly reduce the initial VM boot time.

Creating a self-encrypting VM image using Fedora Rawhide

There are many great tools for creating OS images out there: ImageBuilder, mkosi, kiwi and so on. For simplicity, and to better illustrate the details, we're doing it manually with the dnf command.

Start by creating an OS tree with a minimal set of packages required:

$ mkdir tree
$ sudo dnf -y --use-host-config --releasever=rawhide --no-gpgchecks --installroot=`pwd`/tree install shim-x64 systemd-container dnf vi kernel-uki-virt
$ sudo dnf -y --use-host-config --releasever=rawhide --no-gpgchecks --installroot=`pwd`/tree clean all

The image uses direct boot from shim to the UKI. Verify the version of the kernel in the image:

$ ls -1 tree/lib/modules
7.2.0-0.rc6.260805gc21bb4193868.50.fc45.x86_64

Put the UKI in the EFI system partition and make shim boot it directly:

$ sudo mkdir -p tree/boot/efi/EFI/Linux
$ sudo cp -a tree/lib/modules/7.2.0-0.rc6.260805gc21bb4193868.50.fc45.x86_64/vmlinuz-virt.efi tree/boot/efi/EFI/Linux/ffffffffffffffffffffffffffffffff-7.2.0-0.rc6.260805gc21bb4193868.50.fc45.x86_64.efi
$ echo "shimx64.efi,UKI,\\EFI\\Linux\\ffffffffffffffffffffffffffffffff-7.2.0-0.rc6.260805gc21bb4193868.50.fc45.x86_64.efi ,Comment" | iconv -t UCS-2 | sudo tee tree/boot/efi/EFI/fedora/BOOTX64.CSV

Create an fstab entry for the EFI system partition:

$ sudo tee tree/etc/fstab << EOF
LABEL=ESP             /boot/efi                   vfat    umask=0077,shortname=winnt 0 2
EOF

Create an empty systemd system extension to trigger the systemd-sysext mechanism mounting writable overlay over /usr partition on boot:

$ mkdir -p sysext/usr/lib/extension-release.d
$ tee sysext/usr/lib/extension-release.d/extension-release.usrrw.sys << EOF
NAME="Usr rw"
VERSION="0.0.1"
SYSEXT_ID="usrrw"
SYSEXT_SCOPE="system"
ID="_any"
EOF

A systemd system extension must be signed. Create a key pair, which will later be added to the SecureBoot database, and generate the extension:

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform PEM -out custom_db.pem
$ sudo mkdir -p tree/var/lib/extensions/
$ sudo systemd-repart -S -s `pwd`/sysext/ --private-key=custom_db.key --certificate=custom_db.pem tree/var/lib/extensions/usrrw.sys.raw

Because we're using the hermetic /usr approach, we need to move everything outside of /usr, which we would like to preserve to a place inside /usr:

$ sudo mkdir tree/usr/root-copy
$ sudo mkdir -p tree/usr/root-copy/boot/efi
$ sudo mv tree/{afs,bin,dev,etc,home,lib,lib64,media,mnt,opt,proc,root,run,srv,sbin,sys,tmp,var} tree/usr/root-copy

Enable autologin to simplify testing the resulting image:

$ sudo mkdir -p tree/usr/lib/systemd/system/serial-getty@ttyS0.service.d/
$ sudo tee tree/usr/lib/systemd/system/serial-getty@ttyS0.service.d/autologin.conf << EOF
[Service]
ExecStart=
ExecStart=-/usr/bin/agetty --noreset --noclear --autologin root --keep-baud 115200,57600,38400,9600 - \${TERM}
EOF

Enable systemd system extensions in mutable mode:

$ sudo tee tree/usr/lib/systemd/sysext.conf << EOF
[SysExt]
Mutable=yes
EOF

Create a systemd-repart configuration, which is going to create a new encrypted partition on the first boot:

$ sudo mkdir -p tree/usr/lib/repart.d
$ sudo tee tree/usr/lib/repart.d/60-root.conf << EOF
[Partition]
Type=root
Label=root
Format=ext4
CopyFiles=/usr/root-copy:/
Encrypt=tpm2
EOF

Create the systemd-repart configuration to create the OS image:

$ mkdir repart.d
$ tee repart.d/40-esp.conf << EOF
[Partition]
Type=esp
Format=vfat
CopyFiles=/boot/efi/EFI:/EFI
CopyFiles=/boot/loader:/loader
SizeMinBytes=512M
SizeMaxBytes=512M
Minimize=off
EOF

$ tee repart.d/50-usr.conf << EOF
[Partition]
Type=usr
CopyFiles=/usr/:/
Verity=data
VerityMatchKey=usr
SizeMinBytes=1512M
SizeMaxBytes=1512M
EOF

$ tee repart.d/60-usr-verity.conf << EOF
[Partition]
Type=usr-verity
Verity=hash
VerityMatchKey=usr
VerityDataBlockSizeBytes=4096
VerityHashBlockSizeBytes=4096
SizeMinBytes=256M
SizeMaxBytes=265M
EOF

Create the OS image

With this configuration, we can create the OS image:

$ sudo systemd-repart --empty=create --size=6G --dry-run=no --definitions=$(pwd)/repart.d/ --copy-source=./tree/ vol.img --json=pretty

The output looks like this:

[
    {
        "type" : "esp",
        "label" : "esp",
                  …
    },
    {
        "type" : "usr-x86-64",
        "label" : "usr-x86-64",
        "uuid" : "aca18d5a-5b08-a808-186c-15684009ebda",
        …
        "roothash" : "aca18d5a5b08a808186c15684009ebda1ba8ea564c896ba41c9b2b87cb79ad0c"
    },
    {
        "type" : "usr-x86-64-verity",
        "label" : "usr-x86-64-verity",
        "uuid" : "1ba8ea56-4c89-6ba4-1c9b-2b87cb79ad0c",
        ...
        "roothash" : "aca18d5a5b08a808186c15684009ebda1ba8ea564c896ba41c9b2b87cb79ad0c"
    }
]

Note the roothash value, which must be passed to the guest kernel. To achieve this, create a UKI command-line extension:

$ sudo mkdir -p tree/boot/loader/addons
$ sudo ukify build --secureboot-private-key=custom_db.key --secureboot-certificate=custom_db.pem --cmdline="usrhash=aca18d5a5b08a808186c15684009ebda1ba8ea564c896ba41c9b2b87cb79ad0c audit=0 rw" --output=tree/boot/loader/addons/usrhash.addon.efi

The generated addon must be included in the image we've already generated. We can use a trick here: Remove the ESP partition from the image, and then run systemd-repart again. This doesn't touch /usr and its dm-verity partition, because those are already present, but ESP is created again and it has the addon in it:

$ sudo sfdisk --delete vol.img 1
$ sudo systemd-repart --dry-run=no --definitions=$(pwd)/repart.d/ --copy-source=./tree/ vol.img --json=pretty

The expected output:

[
    {
        "type" : "esp",
        "label" : "esp",
        "uuid" : "00ad5db5-08e0-422d-b0b1-bf9abff343d7",
        …
        "activity" : "create",
        "roothash" : null
    },
    {
        "type" : "usr-x86-64",
        "label" : "usr-x86-64",
        …
        "activity" : "unchanged",
        "roothash" : "TBD"
    },
    {
        "type" : "usr-x86-64-verity",
        "label" : "usr-x86-64-verity",
        …
        "activity" : "unchanged",
        "roothash" : "TBD"
    }
]

Run the generated image with QEMU

To run the image and verify that it works, use QEMU/KVM. The image must run with SecureBoot enabled. UKI command-line extensions are accepted regardless of the signature when SecureBoot is off, but systemd system extension must always pass signature check and the SecureBoot database is a straightforward way to provide trusted keys to the kernel.

First, generate the UEFI variable store with our custom certificate included in it:

$ virt-fw-vars -i /usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2 --add-db `uuidgen --random` custom_db.pem -o OVMF_VARS_4M.qcow2

Make the disk image accessible by the current user so the QEMU process doesn't have to run as root:

$ sudo chown $(whoami):$(id -gn) vol.img

The VM requires a vTPM to seal the encrypted volume key, so use swtpm as an emulator:

$ mkdir tpm2
$ swtpm socket -d --tpm2 --tpmstate dir=`pwd`/tpm2 --ctrl type=unixio,path=`pwd`/tpm2.sock

Start QEMU:

$ qemu-system-x86_64 -smp 4 -machine q35,smm=on,accel=kvm,kernel-irqchip=split -global driver=cfi.pflash01,property=secure,value=on -cpu host -drive id=drive_image2,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=vol.img -device virtio-blk-pci,id=image2,drive=drive_image2,bootindex=3,bus=pcie.0,addr=0x8 -drive file=/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2,if=pflash,format=qcow2,readonly=on,unit=0 -drive file=OVMF_VARS_4M.qcow2,if=pflash,format=qcow2,unit=1 -device ahci,id=ahci0 -vnc :0 -vga std -m 8G -boot menu=on -serial stdio -chardev socket,id=chrtpm,path=`pwd`/tpm2.sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0

The guest boots and automatically logs in:

…
[  OK  ] Finished kernel-bootcfg-boot-successful.service - UKI Successful Boot.

FedoraFedora Linuxnbsp;Linux 45 (Rawhide Prerelease)
Kernel 7.2.0-0.rc6.260805gc21bb4193868.50.fc45.x86_64 on x86_64 (ttyS0)

fedora login: root (automatic login)

[root@fedora ~]#

Verify that the VM's storage is encrypted:

# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0                   7:0    0    1M  1 loop  
├─loop0p1             259:0    0  504K  1 part  
│ └─loop0p1-11-verity 252:2    0    4K  1 crypt 
├─loop0p2             259:1    0  508K  1 part  
│ └─loop0p1-11-verity 252:2    0    4K  1 crypt 
└─loop0p3             259:2    0   16K  1 part  
sr0                    11:0    1 1024M  1 rom   
vda                   253:0    0    6G  0 disk  
├─vda1                253:1    0  512M  0 part  /boot/efi
├─vda2                253:2    0  1.5G  0 part  
│ └─usr               252:0    0  1.5G  1 crypt /usr
├─vda3                253:3    0  265M  0 part  
│ └─usr               252:0    0  1.5G  1 crypt /usr
└─vda4                253:4    0  3.8G  0 part  
  └─root              252:1    0  3.7G  0 crypt /

Verify that the VM's /usr partition is read-write:

# touch my-file-in-usr /usr/

Reboot the image to verify that the changes to files in /usr, /etc, and /var persist after reboot.

Current limitations

The suggested solution has a few shortcomings:

  • Because we are only creating a single overlay partition, only trivial storage configurations are supported. In particular, creating separate /home, /var, and other partitions require modifications to the systemd-repart configuration used in this article.
  • There is no proof of the origin of the encrypted overlay partition. A malicious host can pre-create an overlay partition and seal the key to it to the guest vTPM (for example, by booting a specially crafted guest operating system with the same vTPM). In this scenario, systemd-repart wouldn't create a new encrypted partition on the first boot, and would use the pre-created one. This can be used to inject arbitrary code inside the guest. There is a systemd feature request to address this problem.
  • The proposed solution does not offer integrity and replay attack protection. It is also susceptible to other confidential storage attack vectors we discussed previously in Confidential virtual machine storage attack scenarios.

This is not an exhaustive list. We'd love to hear your thoughts and proposals, so don't hesitate to reach out!

Learn more