Build trusted Python containers with Project Hummingbird and Calunga

Project Hummingbird and Calunga are open source projects that offer the beginnings of a secure software supply chain. These projects help open source and commercial developers build application containers with confidence that they are not shipping malicious content.

This article describes how to use container images produced by Project Hummingbird with libraries produced by the Calunga project to build application containers free of code from unknown origins. You can use these tools to deliver more secure container images for your applications.

Red Hat supports container images produced by Project Hummingbird as part of Red Hat Hardened Images. Red Hat also supports libraries from the Calunga project through the trusted libraries feature in Red Hat Advanced Developer Suite. You can redistribute container images that you build using these tools and run them on any platform that supports the Open Container Initiative (OCI) standard—for example, on Ubuntu Linux, Windows, and macOS computers running Podman or Docker.

You are not required to buy a Red Hat product to use Red Hat Hardened Images and trusted libraries, or to redistribute the containers you build. However, if you have valid subscriptions for Red Hat products such as Red Hat Enterprise Linux, Red Hat OpenShift, or Red Hat Advanced Developer Suite, you receive support for your base images and libraries under your existing support coverage.

Prepare your developer environment

This is the second part of a series about using container images from Project Hummingbird. Part 1 describes how to use Project Hummingbird to run databases and web servers and explores its distroless nature. You can understand this article without reading the first part. This installment focuses on using runtime and builder images from Project Hummingbird for different programming languages, using Python as an example.

You can find the code for this article on GitHub. I tested these commands on my Fedora Linux laptop using the podman command in rootless mode. You can easily adapt these steps for other Linux distributions, Windows, macOS, and Docker.

If you prefer a graphical interface over a command shell, try Podman Desktop. It runs on all major operating systems and includes the Podman container engine.

A security-focused foundation for your supply chain

Like it or not, applications are often the most vulnerable part of an organization's information security chain. In the old times, developers focused primarily on security vulnerabilities in dependencies, such as libraries prone to buffer overflow issues. Updating the copy of the library included within an application container would usually resolve the issue. This process is similar to updating an operating system's libraries and binaries to fix issues at the server or workstation machine level.

Monitoring for Common Vulnerabilities and Exposures (CVEs) and patches from your preferred libraries is no longer enough. Malicious actors now target the communities that produce those libraries and the public infrastructure that distributes them. You might download a library that someone has tampered with. This can happen whether the compromised copy comes from an official community repository or a public mirror. The file could even come from your internal corporate servers. When you include that compromised code in your application, it can harm your organization, your customers, and your users.

Red Hat Hardened Images provides base container images, while trusted libraries provides trusted dependency libraries. Red Hat builds both using a strict, security-focused software development lifecycle. These artifacts meet SLSA build level 3 standards and offer fast resolution times for known CVEs by closely following upstream repositories. Red Hat hosts these artifacts in a monitored, security-focused content distribution network.

A Python example

All sample code for this article is available in a public GitHub repository. To follow along, clone the repository and navigate into the directory.

$ git clone https://github.com/flozanorht/python-hello.git

The repository contains two directories. Start in the single-stage directory; we will switch to the multi-stage directory later.

$ cd python-hello/single-stage

This very simple Python application uses the Flask framework. It returns the version of the Python runtime and the Flask framework as a JSON response. This example is not production-ready because it enables debugging and does not use a production-grade web server.

Listing 1: app/app.py 

import sys
import flask
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/')
def hello():
    return jsonify({
        "python_version": sys.version.split()[0],
        "flask_version": flask.__version__
    })
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)

The following Containerfile packages the application as a container image using public repositories. This Containerfile is representative of recommended practices for Python containers:

• It uses the slim container image variant, which provides a smaller runtime for applications that do not require native code libraries.
• It runs the application as a non-root user to improve security.

Some developers might argue that there's no reason to use Python virtual environments in a container, but you will also find many others favor their use. In this case, a virtual environment simplifies the transition to multi-stage builders later in this article.

Listing 2: Containerfile.dockerhub

FROM python:3.12-slim
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt 
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser
COPY app .
EXPOSE 5000
CMD ["python", "app.py"]

Build the container image using Podman:

$ podman build --no-cache -t flask-app -f Containerfile.dockerhub

Start a test container using your new image:

$ podman run -d -p 127.0.0.1:5000:5000 --name my-flask-app flask-app

Finally, test your application container using a web browser or a curl command:

$ curl 127.0.0.1:5000

The output should look similar to this:

{
  "flask_version": "3.1.3",
  "python_version": "3.12.13"
}

When you're done, stop and delete your test container. You will reuse this container name to test the different variations of your application container image.

$ podman stop my-flask-app ; podman rm my-flask-app

Using base images from Project Hummingbird

Adapting the previous example for Project Hummingbird requires more than replacing the name of the base image in the FROM instruction. While Project Hummingbird maintains compatibility with popular community images, it also minimizes and hardens its images. These security-focused optimizations might require changes to your Containerfile. For example:

• Use the array syntax for RUN instructions. Project Hummingbird images usually do not include a shell, which the string syntax requires.
• Switch to the root user (USER 0) before performing operation that require elevated privileges. Project Hummingbird images do not run as root by default.

We also recommend following these conventions:

• Follow Fedora Linux conventions for paths and command names—for example, use python3 for the Python interpreter.
• Use Project Hummingbird conventions for environment labels, ports, variables, users, and paths. For example, use the CONTAINER_DEFAULT_USER variable, which the base images set as the default user.

Listing 3: single-stage/Containerfile.hummingbird

FROM registry.access.redhat.com/hi/python:3.12
USER 0
RUN [ "python3", "-m", "venv", "/opt/venv" ]
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY app/requirements.txt .
RUN [ "pip3", "install", "--no-cache-dir", "-r", "requirements.txt" ]
USER ${CONTAINER_DEFAULT_USER}
COPY app .
EXPOSE 5000
CMD ["python3", "app.py"]

To build and test your Project Hummingbird application container, use the same commands as the previous example but replace the Containerfile name.

$ podman build --no-cache -t flask-app -f Containerfile.hummingbird
$ podman run -d -p 127.0.0.1:5000:5000 --name my-flask-app flask-app
$ curl 127.0.0.1:5000

When you're satisfied with your tests, stop and delete your application container. You will reuse this container name for other versions of the application image.

$ podman stop my-flask-app ; podman rm my-flask-app

Using trusted libraries

Building an application container from a secure base image while adding unverified dependencies can result in a vulnerable container. Let's modify the previous example to use dependencies from the trusted libraries index.

The only difference between Containerfile.hummingbird and Containerfile.calunga is the use of a mount secret in the RUN instruction that runs pip:

RUN --mount=type=secret,id=pip.conf,target=/opt/venv/pip.conf [ "pip3", "install", "--no-cache-dir", "-r", "requirements.txt" ]

That secret provides a configuration for pip to use the trusted libraries component. But first, you must create the configuration file that contains your personal authentication credentials.

You need a valid Red Hat account. You can use an account with a no-cost Red Hat Developer Subscription for Individuals. We recommend that you create a service account to avoid exposing your credentials in pip configuration files or CI/CD pipeline settings.

Log in to Red Hat Developer or register for an account. Then, access the Red Hat Customer Portal to create a service account. You might need to log in to the Customer Portal again using your Red Hat Developer account.

In your local repository clone, copy the template file pip.conf.orig to pip.conf. Edit the file to include the username and the very long password for your service account. Then build your container image using the pip.conf file as a secret mounted into the build container.

$ podman build --no-cache -t flask-app --secret id=pip.conf,src=pip.conf -f Containerfile.calunga

As your build progresses, notice that it downloads Python wheels from packages.redhat.com. The build command should produce output similar to the following snippet:

...
STEP 7/11: RUN --mount=type=secret,id=pip.conf,target=/opt/venv/pip.conf [ "pip3", "install", "--no-cache-dir", "-r", "requirements.txt" ]
Looking in indexes: https://5346907%7Crhtl:****@packages.redhat.com/trusted-libraries/python/
Collecting Flask>=3.0.0 (from -r requirements.txt (line 1))
  Downloading https://packages.redhat.com/api/pulp-content/public-trusted-libraries/main/flask-3.1.3-0-py3-none-any.whl.metadata (3.2 kB)
…

You will see similar “Downloading” messages for each application dependency.

If all went well, test your new container image:

$ podman run -d -p 127.0.0.1:5000:5000 --name my-flask-app flask-app
$ curl 127.0.0.1:5000

While this version of your application container is still running, you can verify that it does not contain a pip.conf file. This confirms that the image does not record your authentication credentials.

$ podman export my-flask-app | tar t | grep pip.conf

As with the previous attempts, stop and delete your application container.

$ podman stop my-flask-app ; podman rm my-flask-app

A multi-stage example

Many developers prefer using multi-stage builds for several reasons:

• Adding build-time dependencies, such as development headers and a C compiler, to interface with processor native code.
• Providing secrets to the build as environment variables. This prevents the secrets from being recorded in the final application image layers.
• Reducing the size of the final application image by using a base image that excludes build-time dependencies and other conveniences that are not necessary to run the application.

Although our simple application does not strictly require a multi-stage build, the following example serves as a template for applications that do.

When using Docker Hub images, a multi-stage build uses the standard Python image for the first stage and the slim variant for the second stage.

In this case, using a Python virtual environment simplifies copying application dependencies between stages.

Listing 3: multi-stage/Containerfile.dockerhub

FROM docker.io/library/python:3.12 AS builder
# Install non-Python dependencies and additional build-time tools
WORKDIR /app
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt 
FROM docker.io/library/python:3.12-slim
WORKDIR /app
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /app
USER appuser
COPY app .
EXPOSE 5000
CMD ["python", "app.py"]

Building and testing the application container is basically the same as the single-stage process. Be sure to navigate to the multi-stage directory.

$ cd ../multi-stage
$ podman build --no-cache -t flask-app -f Containerfile.dockerhub
$ podman run -d -p 127.0.0.1:5000:5000 --name my-flask-app flask-app
$ curl 127.0.0.1:5000
$ podman stop my-flask-app ; podman rm my-flask-app

For Project Hummingbird, use the builder image variant for the first stage and the regular image for the second stage. The following example also uses the Calunga index during the first stage.

Listing 4: multi-stage/Containerfile.calunga

FROM registry.access.redhat.com/hi/python:3.12-builder AS builder
USER 0
# Install non-Python dependencies and additional build-time tools
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
WORKDIR /app
COPY app/requirements.txt .
RUN --mount=type=secret,id=pip.conf,target=/opt/venv/pip.conf pip3 install --no-cache-dir -r requirements.txt
USER ${CONTAINER_DEFAULT_USER}
FROM registry.access.redhat.com/hi/python:3.12
USER 0
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1
USER ${CONTAINER_DEFAULT_USER}
WORKDIR /app
COPY app .
EXPOSE 5000
CMD ["python3", "app.py"]

You can use the string syntax for RUN instructions in the first stage because Project Hummingbird builder images include a shell. This convenience makes multi-stage builds a popular choice for developers using Project Hummingbird.

To build and test this image, copy the pip.conf file from the previous example using trusted libraries. Then, run the podman and curl commands:

$ cp ../single-stage/pip.conf .
$ podman build --no-cache -t flask-app --secret id=pip.conf,src=pip.conf -f Containerfile.calunga
$ podman run -d -p 127.0.0.1:5000:5000 --name my-flask-app flask-app
$ curl 127.0.0.1:5000
$ podman stop my-flask-app ; podman rm my-flask-app

Continuing your security journey

Using secure base images and trusted dependency libraries is an important first step for your software supply chain. This is a straightforward step that does not require specialized tools like security scanners or sophisticated processes to ensure the provenance and attestation of software artifacts.

The trusted libraries component currently offers dependencies for Python only. We plan to expand coverage to other programming language runtimes that Project Hummingbird serves, such as Node.js and Java.

Project Hummingbird and trusted libraries do not prevent vulnerabilities in your own application code. These tools also do not prevent tampering with application container images stored on internal and public container registries.

The open source project Konflux provides many of the upstream components of Red Hat Advanced Developer Suite. It includes the tools required to implement a security-focused software development lifecycle that creates and signs SBOMs and container images while running security scanners.

Many thanks to Luiz Carvalho and Shane Boulden for their review of this article.