Security-conscious organizations are accustomed to using digital signatures to validate application content from the Internet. A common example is RPM package signing. Red Hat Enterprise Linux (RHEL) validates signatures of RPM packages by default.
In the container world, a similar paradigm should be adhered to. In fact, all container images from Red Hat have been digitally signed and have been for several years. Many users are not aware of this because early container tooling was not designed to support digital signatures.
In this article, I'll demonstrate how to configure a container engine to validate signatures of container images from the Red Hat registries for increased security of your containerized applications.
In the lack of widely accepted standards, Red Hat designed a simple approach to provide security to its customers. This approach is based on detached signatures served by a standard HTTP server. The Linux container tools (Podman, Skopeo, and Buildah) have built-in support for detached signatures, as well as the CRI-O container engine from Kubernetes and the Red Hat OpenShift Container Platform.
Configuring Linux container tools to check image signatures
Configuring Linux container tools to only run container images that pass signature checking is a two-step process:
- Create a YAML file under
/etc/containers/registries.dthat specifies the location of detached signatures for a given registry server. - Add an entry to
/etc/containers/policy.jsonthat specifies the public GPG key that validates signatures of a given registry server.
Red Hat stores signatures for its container images at https://access.redhat.com/webassets/docker/content/sigstore. Red Hat signs its images with the same GPG key it uses to sign RPM packages. All Red Hat Enterprise Linux (RHEL) systems ship with Red Hat’s RPM public key at /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release.
With the information above, you can create the /etc/containers/registries.d/redhat.yaml file with the following content:
docker: registry.access.redhat.com: sigstore: https://access.redhat.com/webassets/docker/content/sigstore
And, you can edit your /etc/containers/policy.json as follows:
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports":
{
"docker-daemon":
{
"": [{"type":"insecureAcceptAnything"}]
},
"docker":
{
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}
]
}
}
}
Testing that Linux container tools refuse images that fail signature check
An easy way to validate your settings is by starting a container using the Red Hat Universal Base Image (UBI). If the following command fails, you may have made a mistake editing your container runtime configuration files:
$ sudo podman run --rm --name test registry.access.redhat.com/ubi8/ubi:8.0-199 date Trying to pull registry.access.redhat.com/ubi8/ubi:8.0-199...Getting image source signatures Checking if image destination supports signatures Copying blob 567fcfc2ff35: 67.77 MiB / 67.77 MiB [=========================] 56s Copying blob 188d0510bf14: 1.48 KiB / 1.48 KiB [===========================] 56s Copying config a73bf97264a0: 4.43 KiB / 4.43 KiB [==========================] 0s Writing manifest to image destination Storing signatures Sat Oct 5 17:24:31 UTC 2019
The output from Podman appears similar to the result that you would get without configuring signature checking. One way to confirm that signatures are actually being validated is to force a validation error by specifying an incorrect GPG key.
If you have the EPEL repository configured on your system, you can edit /etc/containers/policy.json as follows to use the GPG key from that repository:
...
"docker":
{
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8"
}
]
}
...
Remove the container image you already downloaded:
$ sudo podman rmi registry.access.redhat.com/ubi8/ubi:8.0-199
And try starting a new container. Now it fails because the container image signature does not match the GPG key:
$ sudo podman run --rm --name test \ registry.access.redhat.com/ubi8/ubi:8.0-199 date Trying to pull registry.access.redhat.com/ubi8/ubi:8.0-199...Failed unable to pull registry.access.redhat.com/ubi8/ubi:8.0-199: unable to pull image: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: ...
The error message is actually much longer and scarier. I only show the first few lines of the message, but it is clear why Podman was unable to pull the image.
After this test, remember to fix your /etc/containers/policy.json so it specifies the correct GPG key for Red Hat container images and RHEL RPM packages.
Validating detached image signatures with rootless containers
Note that my previous commands use sudo to run podman. This is how you would perform these tasks on RHEL and CentOS up to 7.6 along with older Fedora releases. Non-RHEL systems will not include the public GPG key to validate RHEL packages. These keys can be downloaded from the Red Hat Customer Portal [1].
If you are on a very recent RHEL, CentOS, or Fedora release, you probably have support for rootless containers [2]. This means you do not need to use sudo anymore. The configuration file changes remain the same.
Mirroring Red Hat container images to a private registry
Many organizations do not allow their servers to download application content directly from the Internet, regardless of whether the content originates from a trusted vendor and is digitally signed. These organizations typically deploy a private registry server internally and require that its servers, and sometimes its developers, to only pull container images from this location.
To emulate that scenario, I will copy the UBI container image to my personal account at Quay.io and configure my container runtime to check image signatures for images stored there. I will store my detached signatures into a local folder to avoid the need to set up an HTTP server.
When you copy container images between registry servers, you cannot just copy their detached signatures, too. These signatures are tied to the full container image reference, which includes the registry name. So, the signatures from Red Hat become invalid for their copies on a private registry.
One way to solve that issue is to use a private GPG key that your organization owns. Then, you copy the public GPG key to any server and developer workstation that needs it. To keep things simple, I will use my personal GPG key pair that I already use to sign email. If it is your first time working with GPG, see the references section [3] for a nice tutorial about generating your GPG key pair.
Fortunately, the signing process is very easy, and it is handled by the same skopeo copy command you would use to mirror the image into your private registry.
Signing container images
Create the /etc/containers/registries.d/quayio.yaml file with the following content, and replace “flozanorht” with your account at Quay.io:
docker: quay.io/flozanorht/ubi: sigstore: file:///var/lib/atomic/sigstore/ sigstore-staging: file:///var/lib/atomic/sigstore/
You need to either give your user permission to write to the /var/lib/atomic/sigstore/ or pick a different folder.
Then, edit your /etc/containers/policy.json file as follows:
...
"docker":
{
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}
],
"quay.io/flozanorht/ubi": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/flozano-pub"
}
]
}
...
To export my GPG public key to /etc/pki/rpm-gpg/flozano-pub, I ran the following command:
$ gpg --export --armor flozano@redhat.com > /tmp/flozano-pub $ sudo mv /tmp/flozano-pub /etc/pki/rpm-gpg/flozano-pub
And, finally, to mirror the UBI image, the skopeo copy command needs the --remove-signatures option. Without it, Skopeo tries to copy the signatures to the destination registry server, and this is only supported on the OpenShift internal registry. Signatures of the source images are still validated during the copy operation.
Next, login to your account at Quay.io and use the skopeo copy command to create your mirror of the UBI container image:
$ podman login -u flozanorht quay.io Password: Login Succeeded! $ skopeo copy --remove-signatures --sign-by flozano@redhat.com \ docker://registry.access.redhat.com/ubi8/ubi:8.0-199 \ docker://quay.io/flozanorht/ubi:8.0-199
Now, as another way to verify that the container runtime is actually checking container image signatures, use the --log-level debug option from podman to start a container using your copy of the UBI image:
$ sudo podman --log-level debug run --rm --name test \ quay.io/flozanorht/ubi:8.0-199 date Trying to pull quay.io/flozanorht/ubi:8.0-199...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000] Using "docker" namespace quay.io/flozanorht/ubi DEBU[0000] Using file:///var/lib/atomic/sigstore/ ... DEBU[0001] GET https://quay.io/v2/flozanorht/ubi/manifests/8.0-199 DEBU[0002] IsRunningImageAllowed for image docker:quay.io/flozanorht/ubi:8.0-199 DEBU[0002] Using transport "docker" specific policy section quay.io/flozanorht/ubi DEBU[0002] Reading /var/lib/atomic/sigstore//flozanorht/ubi@sha256=c318fd9549dda67f3a1b3aa19b55b26b9dd42f597c3f2ee4181f0d9de16226de/signature-1 DEBU[0002] Reading /var/lib/atomic/sigstore//flozanorht/ubi@sha256=c318fd9549dda67f3a1b3aa19b55b26b9dd42f597c3f2ee4181f0d9de16226de/signature-2 DEBU[0002] Requirement 0: allowed DEBU[0002] Overall: allowed Getting image source signatures ... DEBU[0002] Started container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f Wed Oct 9 03:41:38 UTC 2019 DEBU[0002] Checking container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f status… DEBU[0002] Cleaning up container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f
It is hard to find the output of the podman command with all the logging around it. The debug messages could be more clear that the signature was valid. However, you can see that it reads the quay.yaml and policy.json configuration files and reads the signatures from /var/lib/atomic/sigstore/ before stating Requirement 0: allowed.
Conclusion
In this article, I showed how anyone can play with container image signatures without a complicated setup. Moving into a production scenario, you would do two things:
- Add the Red Hat terms-based registry at registry.redhat.io to your configuration.
- Work on a script that lists all tags of the images you want to mirror and then copy each of them. Skopeo is smart enough to not copy again images and layers you already have on your private registry.
RHEL7 users can manage the signature-related files in /etc/containers using the atomic command. RHEL8 can do the same using the podman image trust command (not to be confused with the podman images command). See references 4, 5, and 6 for more information.
If you are new to the Linux container tools (Podman, Buildah, and Skopeo), see references 7 and 8 below.
References
1. Product Signing Keys on the Red Hat Customer Portal
2. Using the rootless containers Tech Preview in RHEL 8.0
4. Signed Images from the Red Hat Container Catalog
5. Verifying image signing for Red Hat Container Registry
6. https://github.com/containers/image/tree/master/docs
Last updated: March 29, 2023