Trust

This lesson is focused on understanding the purpose of container registries,how they work, and how you can discover the trustworthiness of an image.

In order to get full benefit from taking this lesson, you need:

• The GitHub command-line interface.
• A copy of the GitHub repository, https://github.com/redhat-developer/demos/intro-both.

By the end of this lesson, you should be able to:

• Evaluate the quality of a container registry.
• Evaluate the quality of a container repository.
• Share your images using public and private registries.

Understand the basics of trust: Quality and provenance

The goal of this exercise is to understand the basics of trust when it comes to registry servers and repositories. This requires quality and provenance, which is just a fancy way of saying:

• You must download a trusted thing
• You must download from a trusted source

Both of these are necessary, but neither alone is sufficient. This has been true since the days of downloading ISO images for Linux distros. Whether evaluating open source libraries or code, prebuilt packages (RPMs or Debs), or container images, we must:

• Determine if we trust the image by evaluating:
  • The quality of the code.
  • The quality of the people and organizations involved in the project.
  • If it has enough history and investment.
  • If it actually works for us.
• Determine if we trust the registry by understanding the quality of its relationship with the trusted project, meaning if we download something from the official GitHub repo, we trust it more than from a fork by user Haxor5579. This is true with ISOs from mirror sites and with image repositories built by people who aren't affiliated with the underlying code or packages.

There are plenty of examples where people ignore one of the above and get hacked. In a previous lesson, we learned how to break the URL down into a registry server, namespace, and repository.

Trusted thing

From a security perspective, it's much better to remotely inspect and determine if we trust an image before we download it, expand it, and cache it in the local storage of our container engine. Every time we download an image and expose it to the graph driver in the container engine, we expose ourselves to potential attack. First, let's do a remote inspection with Skopeo. We can't do that with Docker because of the client/server nature.

skopeo inspect docker://registry.fedoraproject.org/fedora

The output will look like this:

{
    "Name": "registry.fedoraproject.org/fedora",
    "Digest": "sha256:d5660c5eb693ef675100ee958a466ff5fa232f2bba906bf348ae2c737eda9c90",
    "RepoTags": [
[79 lines removed here]
    ],
    "Created": "2025-04-24T06:49:25.660023937Z",
    "DockerVersion": "",
    "Labels": {
        "io.buildah.version": "1.39.3",
        "license": "MIT",
        "name": "fedora",
        "org.opencontainers.image.license": "MIT",
        "org.opencontainers.image.name": "fedora",
        "org.opencontainers.image.url": "https://fedoraproject.org/",
        "org.opencontainers.image.vendor": "Fedora Project",
        "org.opencontainers.image.version": "42",
        "vendor": "Fedora Project",
        "version": "42"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:3f7284d6b026d411550d7e371c47a8ad1b5726169f9c06995d1ef06b8eb8290e"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "Digest": "sha256:3f7284d6b026d411550d7e371c47a8ad1b5726169f9c06995d1ef06b8eb8290e",
            "Size": 61312774,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/bin:/usr/bin",
        "container=oci"
    ]
}

Examine the JSON. There's really nothing in there that helps us determine if we trust this repository. It says it was created by the Fedora project (vendor, Fedora Project), but we have no idea if that is true. We have to move on to verifying that we trust the source, then determine if we trust the thing.

Trusted source

There's a lot of talk about image signing, but the reality is, most people are not verifying container images with signatures. What they are actually doing is relying on SSL to determine that they trust the source, then inferring that they trust the container image. Let's use this knowledge to do a quick evaluation of the official Fedora registry.

Run the following command:

curl -I https://registry.fedoraproject.org

If you are on a Fedora or a Red Hat Enterprise Linux (RHEL) box with the right keys, the output looks similar to this:

HTTP/2 200
date: Thu, 25 Apr 2019 17:50:25 GMT
server: Apache/2.4.39 (Fedora)
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: same-origin
last-modified: Thu, 25 Apr 2019 17:25:08 GMT
etag: "1d6ab-5875e1988dd3e"
accept-ranges: bytes
content-length: 120491
apptime: D=280
x-fedora-proxyserver: proxy10.phx2.fedoraproject.org
x-fedora-requestid: XMHzYeZ1J0RNEOvnRANX3QAAAAE
content-type: text/html

We can discern that the certificate is valid and managed by Red Hat (which helps a bit) by running a command.

If using Bash, run the following command:

curl 2>&1 -kvv https://registry.fedoraproject.org | grep subject

If using PowerShell, run the following command and check the Subject property of the output:

iex (iwr https://raw.githubusercontent.com/redhat-developer-demos/intro-both/refs/heads/main/checkcert.ps1).Content

Think carefully about what we just did. Even visually validating the certificate gives us some minimal level of trust in this registry server. In a real-world scenario, remember that it's the container engine's job to check these certificates. That means that systems administrators need to distribute the appropriate CA certificates in production.

Now that we have inspected the certificate, we can safely pull the trusted repository from the trusted registry server because we trust the Fedora project to build it correctly, and we know it is managed by Fedora/Red Hat.

podman pull registry.fedoraproject.org/fedora

Now let's move on to evaluate some trickier repositories and registry servers.

Evaluate trust: Images and registry servers

Images

First, let's start with what we already know: There is often a fully-functioning Linux distro inside a container image. That's because it's useful to leverage existing packages and the dependency tree already created for it. This is true whether running on bare metal, in a virtual machine (VM), or in a container image. It's also important to consider the quality, frequency, and ease of consuming updates in the container image.

To analyze the quality, we are going to leverage existing tools, which is another advantage of consuming a container image based on a Linux distro. To demonstrate, let's examine images from four different Linux distros: 

• CentOS 

• Fedora 

• Ubuntu 

• Red Hat Enterprise Linux 

Each will provide differing levels of information.

CentOS

podman run -it quay.io/centos/centos yum updateinfo

CentOS does not provide errata for package updates, so this command will not show any information. This makes it difficult to map CVEs to RPM packages. This, in turn, makes it difficult to update the packages that are affected by a CVE. Finally, this lack of information makes it difficult to score a container image for quality. A basic workaround is to just update everything, but even then, you are not 100% sure which CVEs you patched.

Fedora

Run the following command:

podman run -it registry.fedoraproject.org/fedora dnf advisory summary

Fedora provides decent metadata about package updates, but does not map them to CVEs either. Results will vary on any given day, but the output often looks like this:

Updating and loading repositories:
 Fedora 42 - x86_64 - Updates                                                                                                                                                         100% |   3.1 MiB/s |   4.6 MiB |  00m01s
 Fedora 42 openh264 (From Cisco) - x86_64                                                                                                                                             100% |  12.3 KiB/s |   6.0 KiB |  00m00s
 Fedora 42 - x86_64                                                                                                                                                                   100% |  14.3 MiB/s |  35.4 MiB |  00m02s
Repositories loaded.
Available advisory information summary:
Security    : 0
  Critical  : 0
  Important : 0
  Moderate  : 0
  Low       : 0
  Other     : 0
Bugfix      : 0
Enhancement : 0
Other       : 0

Ubuntu

Run the following command:

podman run -it docker.io/ubuntu:trusty-20170330 /bin/bash -c "apt-get update && apt list --upgradable"

Ubuntu provides information at a similar quality to Fedora, but again does not map updates to CVEs easily. The results for this specific image should always be the same because we are purposefully pulling an old tag for demonstration purposes.

Red Hat Enterprise Linux

Run the following command:

podman run -it registry.access.redhat.com/ubi9/ubi yum updateinfo –info

Regretfully, we do not have the active Red Hat subscriptions necessary to analyze the Red Hat Universal Base Image (UBI) on the command line, but the output would resemble the following if it ran on Red Hat Enterprise Linux or Red Hat OpenShift:

Updating Subscription Management repositories.
subscription-manager is operating in container mode.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                  	17 MB/s |  52 MB 	00:03    
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                               	22 MB/s |  53 MB 	00:02    
Red Hat Universal Base Image 9 (RPMs) - BaseOS                                                                        	959 kB/s | 528 kB 	00:00    
Red Hat Universal Base Image 9 (RPMs) - AppStream                                                                     	3.4 MB/s | 2.3 MB 	00:00    
Red Hat Universal Base Image 9 (RPMs) - CodeReady Builder                                                             	585 kB/s | 282 kB 	00:00    
===============================================================================
  Moderate: expat security update
===============================================================================
  Update ID: RHSA-2025:3531
   	Type: security
	Updated: 2025-04-02 13:36:22
   	Bugs: 2310137 - CVE-2024-8176 libexpat: expat: Improper Restriction of XML Entity Expansion Depth in libexpat
   	CVEs: CVE-2024-8176
Description: Expat is a C library for parsing XML documents.
       	:
       	: Security Fix(es):
       	:
       	: * libexpat: expat: Improper Restriction of XML Entity Expansion Depth in libexpat (CVE-2024-8176)
       	:
       	: For more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section.
   Severity: Moderate

Notice the RHSA information. This indicates the errata and its level of importance. This errata can be used to map the update to a particular CVE, which gives you and your security team the confidence that a container image is patched for any particular CVE. 

Even without a Red Hat subscription, we can analyze the quality of a Red Hat image by looking at the Red Hat Container Catalog and using the Container Health Index. Click Red Hat Universal Base Image 9 - Red Hat Ecosystem Catalog then Learn more about the Health Index (Figure 1).

You will see a window like this:

Evaluate registries

Now that we have taken a look at several container images, we are going to start to look at where they came from and how they were built by evaluating four registry servers:

• Fedora

• DockerHub

• Bitnami 

• Red Hat Container Catalog

Fedora registry: https://quay.io/organization/fedora The Fedora registry provides a very basic experience. You know that it is operated by the Fedora project, so the security should be pretty similar to the ISOs you download. That said, there are no older versions of images, and there is really no stated policy about how often the images are patched, updated, or released.

DockerHub: https://hub.docker.com/_/nginx DockerHub provides official images for a lot of different pieces of software, including Ubuntu, WordPress, and PHP. That said, there really isn't a standard definition for what official means. Each repository appears to have its own processes, rules, timelines, lifecycles, and testing. There is also no shared understanding of what official images provide to an end user. Users must evaluate each repository for themselves and determine whether they trust it's connected to the upstream project in a meaningful way.

Bitnami: https://bitnami.com/containers Similar to DockerHub, there is not a lot of information linking these repositories to the upstream projects in a meaningful way. There is not even a clear understanding of what tags are available or should be used. Again, minimal policy information pretty much leaves users to sift through GitHub repositories to have any understanding of how they are built or if there are any lifecycle guarantees about versions. You are left to just trust that Bitnami builds containers the way you want them.

Red Hat Container Catalog: https://catalog.redhat.com/search?searchType=containers The Red Hat Container Catalog is set up in a completely different way than almost every other registry server. There is a tremendous amount of information about each repository. Poke around and notice how this particular image has a warning associated. For this exercise, we are purposefully looking at an older image with known vulnerabilities. That's because container images age like cheese, not like wine. Trust is temporal, and older container images age just like servers that are rarely or never patched.

Now take a look at the Container Health Index scoring for each available tag. The newer the tag, the better the letter grade. The Red Hat Container Catalog and Container Health Index clearly show that newer images have fewer vulnerabilities and hence have a better letter grade. To fully understand the scoring criteria, check out the Knowledge Base Article. This is a unique capability provided by the Red Hat Container Catalog because container image errata tie container images to CVEs.

Knowing what you know now:

• How would you analyze these container repositories to determine if you trust them?
• How would you rate your trust in these registries?
• Is the reputation of the brand enough? Tooling? Lifecycle? Quality?
• How would you analyze repositories and registries to meet the needs of your company?

These questions seem easy, but they're really not. It prompts you to revisit what it means to trust a container registry and repository.

Analyze storage and graph drivers

In this lesson, we are going to focus on how container engines cache repositories on the container host. There is a little-known fact that whenever you pull a container image, each layer is cached locally and mapped into a shared filesystem, typically overlay2 or devicemapper. 

This has a few implications. First, this means that caching a container image locally has historically been a root operation. Second, if you pull an image or commit a new layer with a password in it, anybody on the system can see it, even if you never push it to a registry server.

Now, let's take a look at the Podman container engine. It pulls OCI-compliant, Docker-compatible images.

If using Bash, run the following command:

podman info  | grep -A4 graphRoot

If using PowerShell, run the following command:

podman info | select-string -Pattern 'graphRoot' -Context 0,4

You might be asking yourself, what is d_type?. Long story short, it's a filesystem option that must be supported for overlay2 to work properly as a backing store for container images and running containers

Conclusion

With Podman, as well as most other container engines on the planet like Docker, image layers are mapped one-for-one to some kind of storage, be it thin snapshots with devicemapper or directories with overlay2.

This has implications for how you move container images from one registry to another. First, you have to pull it and cache it locally, then tag it with the URL, namespace, repository, and tag that you want in the new registry. Finally, you have to push it. This is a convoluted mess, and in a later lesson, we will investigate a tool called Skopeo that makes this much easier.

It is a journey, and we are always happy to help. If you want more options, consider the following learning paths:

• Install Podman Desktop and connect it to your Developer Sandbox
• Download and install the Red Hat OpenShift CLI
• Move your Developer Sandbox objects to another cluster

For now, you understand enough about registry servers, repositories, and how images are cached locally. The next engine addresses container engineers and the Linux kernel..