Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Verifying signatures of Red Hat container images

 

October 29, 2019
Fernando Lozano
Related topics:
ContainersLinuxKubernetesSecurity
Related products:
Red Hat OpenShiftRed Hat OpenShift Container Platform

Share:

    Security-conscious organizations are accustomed to using digital signatures to validate application content from the Internet. A common example is RPM package signing. Red Hat Enterprise Linux (RHEL) validates signatures of RPM packages by default.

    In the container world, a similar paradigm should be adhered to. In fact, all container images from Red Hat have been digitally signed and have been for several years. Many users are not aware of this because early container tooling was not designed to support digital signatures.

    In this article, I'll demonstrate how to configure a container engine to validate signatures of container images from the Red Hat registries for increased security of your containerized applications.

    In the lack of widely accepted standards, Red Hat designed a simple approach to provide security to its customers. This approach is based on detached signatures served by a standard HTTP server. The Linux container tools (Podman, Skopeo, and Buildah) have built-in support for detached signatures, as well as the CRI-O container engine from Kubernetes and the Red Hat OpenShift Container Platform.

    Configuring Linux container tools to check image signatures

    Configuring Linux container tools to only run container images that pass signature checking is a two-step process:

    1. Create a YAML file under /etc/containers/registries.d that specifies the location of detached signatures for a given registry server.
    2. Add an entry to /etc/containers/policy.json that specifies the public GPG key that validates signatures of a given registry server.

    Red Hat stores signatures for its container images at https://access.redhat.com/webassets/docker/content/sigstore. Red Hat signs its images with the same GPG key it uses to sign RPM packages. All Red Hat Enterprise Linux (RHEL) systems ship with Red Hat’s RPM public key at /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release.

    With the information above, you can create the /etc/containers/registries.d/redhat.yaml file with the following content:

    docker:
      registry.access.redhat.com:
        sigstore: https://access.redhat.com/webassets/docker/content/sigstore

    And, you can edit your /etc/containers/policy.json as follows:

    {
      "default": [
        {
          "type": "insecureAcceptAnything"
        }
      ],
      "transports":
        {
          "docker-daemon":
            {
              "": [{"type":"insecureAcceptAnything"}]
            },
          "docker":
            {
              "registry.access.redhat.com": [
                {
                  "type": "signedBy",
                  "keyType": "GPGKeys",
                  "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
                }
              ]
            }
        }
    }

    Testing that Linux container tools refuse images that fail signature check

    An easy way to validate your settings is by starting a container using the Red Hat Universal Base Image (UBI). If the following command fails, you may have made a mistake editing your container runtime configuration files:

    $ sudo podman run --rm --name test registry.access.redhat.com/ubi8/ubi:8.0-199 date
    Trying to pull registry.access.redhat.com/ubi8/ubi:8.0-199...Getting image source signatures
    Checking if image destination supports signatures
    Copying blob 567fcfc2ff35: 67.77 MiB / 67.77 MiB [=========================] 56s
    Copying blob 188d0510bf14: 1.48 KiB / 1.48 KiB [===========================] 56s
    Copying config a73bf97264a0: 4.43 KiB / 4.43 KiB [==========================] 0s
    Writing manifest to image destination
    Storing signatures
    Sat Oct  5 17:24:31 UTC 2019

    The output from Podman appears similar to the result that you would get without configuring signature checking. One way to confirm that signatures are actually being validated is to force a validation error by specifying an incorrect GPG key.

    If you have the EPEL repository configured on your system, you can edit /etc/containers/policy.json as follows to use the GPG key from that repository:

    ...
          "docker":
            {
              "registry.access.redhat.com": [
                {
                  "type": "signedBy",
                  "keyType": "GPGKeys",
                  "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8"
                }
              ]
            }
    ...

    Remove the container image you already downloaded:

    $ sudo podman rmi registry.access.redhat.com/ubi8/ubi:8.0-199

    And try starting a new container. Now it fails because the container image signature does not match the GPG key:

    $ sudo podman run --rm --name test \
    registry.access.redhat.com/ubi8/ubi:8.0-199 date
    Trying to pull registry.access.redhat.com/ubi8/ubi:8.0-199...Failed
    unable to pull registry.access.redhat.com/ubi8/ubi:8.0-199: unable to pull image: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: ...

    The error message is actually much longer and scarier. I only show the first few lines of the message, but it is clear why Podman was unable to pull the image.

    After this test, remember to fix your /etc/containers/policy.json so it specifies the correct GPG key for Red Hat container images and RHEL RPM packages.

    Validating detached image signatures with rootless containers

    Note that my previous commands use sudo to run podman. This is how you would perform these tasks on RHEL and CentOS up to 7.6 along with older Fedora releases. Non-RHEL systems will not include the public GPG key to validate RHEL packages. These keys can be downloaded from the Red Hat Customer Portal [1].

    If you are on a very recent RHEL, CentOS, or Fedora release, you probably have support for rootless containers [2]. This means you do not need to use sudo anymore. The configuration file changes remain the same.

    Mirroring Red Hat container images to a private registry

    Many organizations do not allow their servers to download application content directly from the Internet, regardless of whether the content originates from a trusted vendor and is digitally signed. These organizations typically deploy a private registry server internally and require that its servers, and sometimes its developers, to only pull container images from this location.

    To emulate that scenario, I will copy the UBI container image to my personal account at Quay.io and configure my container runtime to check image signatures for images stored there. I will store my detached signatures into a local folder to avoid the need to set up an HTTP server.

    When you copy container images between registry servers, you cannot just copy their detached signatures, too. These signatures are tied to the full container image reference, which includes the registry name. So, the signatures from Red Hat become invalid for their copies on a private registry.

    One way to solve that issue is to use a private GPG key that your organization owns. Then, you copy the public GPG key to any server and developer workstation that needs it. To keep things simple, I will use my personal GPG key pair that I already use to sign email. If it is your first time working with GPG, see the references section [3] for a nice tutorial about generating your GPG key pair.

    Fortunately, the signing process is very easy, and it is handled by the same skopeo copy command you would use to mirror the image into your private registry.

    Signing container images

    Create the /etc/containers/registries.d/quayio.yaml file with the following content, and replace “flozanorht” with your account at Quay.io:

    docker:
      quay.io/flozanorht/ubi:
        sigstore: file:///var/lib/atomic/sigstore/
        sigstore-staging: file:///var/lib/atomic/sigstore/

    You need to either give your user permission to write to the /var/lib/atomic/sigstore/ or pick a different folder.

    Then, edit your /etc/containers/policy.json file as follows:

    ...
          "docker":
            {
              "registry.access.redhat.com": [
                {
                  "type": "signedBy",
                  "keyType": "GPGKeys",
                  "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
                }
              ],
              "quay.io/flozanorht/ubi": [
                {
                  "type": "signedBy",
                  "keyType": "GPGKeys",
                  "keyPath": "/etc/pki/rpm-gpg/flozano-pub"
                }
              ]
            }
    ...

    To export my GPG public key to /etc/pki/rpm-gpg/flozano-pub, I ran the following command:

    $ gpg --export --armor flozano@redhat.com > /tmp/flozano-pub
    $ sudo mv /tmp/flozano-pub /etc/pki/rpm-gpg/flozano-pub

    And, finally, to mirror the UBI image, the skopeo copy command needs the --remove-signatures option. Without it, Skopeo tries to copy the signatures to the destination registry server, and this is only supported on the OpenShift internal registry. Signatures of the source images are still validated during the copy operation.

    Next, login to your account at Quay.io and use the skopeo copy command to create your mirror of the UBI container image:

    $ podman login -u flozanorht quay.io
    Password:
    Login Succeeded!
    $ skopeo copy --remove-signatures --sign-by flozano@redhat.com \
    docker://registry.access.redhat.com/ubi8/ubi:8.0-199 \
    docker://quay.io/flozanorht/ubi:8.0-199

    Now, as another way to verify that the container runtime is actually checking container image signatures, use the --log-level debug option from podman to start a container using your copy of the UBI image:

    $ sudo podman --log-level debug run --rm --name test \
    quay.io/flozanorht/ubi:8.0-199 date
    Trying to pull quay.io/flozanorht/ubi:8.0-199...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
    DEBU[0000]  Using "docker" namespace quay.io/flozanorht/ubi
    DEBU[0000]   Using file:///var/lib/atomic/sigstore/
    ...
    DEBU[0001] GET https://quay.io/v2/flozanorht/ubi/manifests/8.0-199
    DEBU[0002] IsRunningImageAllowed for image docker:quay.io/flozanorht/ubi:8.0-199
    DEBU[0002]  Using transport "docker" specific policy section quay.io/flozanorht/ubi
    DEBU[0002] Reading /var/lib/atomic/sigstore//flozanorht/ubi@sha256=c318fd9549dda67f3a1b3aa19b55b26b9dd42f597c3f2ee4181f0d9de16226de/signature-1
    DEBU[0002] Reading /var/lib/atomic/sigstore//flozanorht/ubi@sha256=c318fd9549dda67f3a1b3aa19b55b26b9dd42f597c3f2ee4181f0d9de16226de/signature-2
    DEBU[0002]  Requirement 0: allowed                     
    DEBU[0002] Overall: allowed                            
    Getting image source signatures
    ...
    DEBU[0002] Started container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f
    Wed Oct  9 03:41:38 UTC 2019
    DEBU[0002] Checking container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f status…
    DEBU[0002] Cleaning up container 5148003a4eebc4ac256add74f127b4993217909189f6571b15fc77e883fbcd4f

    It is hard to find the output of the podman command with all the logging around it. The debug messages could be more clear that the signature was valid. However, you can see that it reads the quay.yaml and policy.json configuration files and reads the signatures from /var/lib/atomic/sigstore/ before stating Requirement 0: allowed.

    Conclusion

    In this article, I showed how anyone can play with container image signatures without a complicated setup. Moving into a production scenario, you would do two things:

    • Add the Red Hat terms-based registry at registry.redhat.io to your configuration.
    • Work on a script that lists all tags of the images you want to mirror and then copy each of them. Skopeo is smart enough to not copy again images and layers you already have on your private registry.

    RHEL7 users can manage the signature-related files in /etc/containers using the atomic command. RHEL8 can do the same using the podman image trust command (not to be confused with the podman images command). See references 4, 5, and 6 for more information.

    If you are new to the Linux container tools (Podman, Buildah, and Skopeo), see references 7 and 8 below.

    References

    1. Product Signing Keys on the Red Hat Customer Portal

    2. Using the rootless containers Tech Preview in RHEL 8.0

    3. Generating a new GPG key

    4. Signed Images from the Red Hat Container Catalog

    5. Verifying image signing for Red Hat Container Registry

    6. https://github.com/containers/image/tree/master/docs

    7. Say “Hello” to Buildah, Podman, and Skopeo

    8. Podman and Buildah for Docker users

    Last updated: March 29, 2023

    Recent Posts

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    • Speech-to-text with Whisper and Red Hat AI Inference Server

    What’s up next?

     

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue