If you've ever watched a container build fail because a package mirror glitched, or wondered whether last month's image really matches what you ship today, then you already know the pain that a hermetic build is meant to solve. For Open Data Hub (ODH) and Red Hat OpenShift AI notebook images, we implemented a simple rule: Nothing downloads during the image build. Dependencies are pinned in lockfiles, prefetched ahead of time, and installed from a local cache while the build runs with no network. The same Containerfile works on a laptop, in GitHub Actions, and on Konflux.
This article tells the story of that shift, why it matters, what surprised us, and where to look if you want to reuse the pattern.
Why "no network during build" matters
A hermetic container build is offline on purpose. Tools like Cachi2 and Hermeto download every RPM, Python wheel, npm package, and Go module before podman or buildah starts. The Containerfile only installs from that cache.
This buys you 3 things:
- Reproducibility: The same lockfiles produce the same dependency tree, even when mirrors drift or floating tags move.
- Auditability: Packages are pinned by URL and checksum. SBOMs can name real ecosystems (
rpm,pip,npm,gomod) instead of opaque tarball URLs. - Compliance: OpenShift AI product builds on Konflux require network isolation and Conforma checks. Upstream ODH images use the same hermetic pipeline. The stricter policy applies on the product path.
For the full architecture diagram and environment matrix, see the hermetic guide in the Open Data Hub repository.
One Containerfile, 3 places to build
The mental model is small:
- Commit lockfiles
- Prefetch into a cache
- Build with the network off
Locally, and in GitHub Actions, we run a shared prefetch script. On Konflux, a Tekton task does the same job. The Containerfile does not care which environment filled the cache. It only installs from it.
Upstream (ODH) and downstream keep separate lock trees. Mixing them fails in subtle ways—as with CentOS and Red Hat Enterprise Linux (RHEL) FIPS package names. If you work with subscribed RHEL bases, start with subscribed builds.
Most Jupyter and runtime images share a repo-root prefetch tree. Codeserver keeps its own, because its dependency set (especially npm) is a different matter.
The design rule that paid off
Early on, we were tempted to just download the binary for awkward tools, like ripgrep releases, Pandoc tarballs, oc mirrors, VS Code marketplace extensions. That works until Conforma asks what those blobs actually are.
At Red Hat, the bar is higher than "it builds offline". Product images must build from source or consume approved dependencies, such as Python wheels published through Artificial Intelligence PC Consortium (AIPCC). A GitHub release tarball prefetched as type: generic may satisfy hermetic networking, but it does not satisfy that packaging expectation.
The rule we settled on: Prefer typed ecosystems (rpm, pip, npm, gomod, or vendored source) over generic URL downloads. Avoid generic prefetch as much as possible.
Whenever we could turn a loose tarball into a first-class package (or an RPM from a known repo), SBOMs got clearer and release checks got quieter. Generic prefetch stayed mainly for things like GPG keys needed to verify prefetched RPMs, not as a dumping ground for binaries. If you truly cannot package a dependency that way, work with ProdSec on an exception. Treat exceptions as temporary bridges, not the default path.
How we generate RPM and Python lockfiles, and how to run prefetch yourself, is detailed in the lockfile generators README.
The hard cases: Codeserver, ripgrep, and Pandoc
Codeserver was the extreme end of the spectrum: A pinned code-server submodule, a large npm graph, URL rewriting into the local cache, then npm ci --offline. If you only read one deep dive, make it the codeserver README.
Ripgrep was a classic Node trap. Upstream @vscode/ripgrep downloads a binary in postinstall, which is fine on the open internet but fatal in a hermetic build. Working with AIPCC, we published a multi-arch ripgrep wheel, prefetched it as pip, pointed the environment at the installed binary, and patched the cached postinstall to copy that binary instead of downloading. The same idea generalizes to a lot of downloads-at-install-time Node packages.
Pandoc followed the same path: Swap a static tarball for a pandoc-rhai pip wheel so the SBOM sees a real Python package. For FIPS and payload-check context, see fips.md.
Partner early on awkward binaries. Having multi-arch wheels before you need them in CI saves weeks of "works on my laptop, fails hermetically."
Passing Conforma without treating exceptions as the plan
On the OpenShift AI path, Conforma looks for hermetic trusted tasks, RPM signatures, multi-arch consistency, SBOM shape, and required labels. Policy lives in places like rhtap-ec-policy and conforma/policy.
Exceptions (including ones coordinated with ProdSec) can bridge a gap. They must not replace fixing packaging shape: Build from source or use approved dependencies, and keep generic prefetch rare. Typed ecosystems beat generic tarballs every time. How we validate locally is detailed in docs/conforma.md.
What we would tell another team
Some advice based on our experience so far:
- Treat hermetic as a packaging practice, not a single pipeline flag.
- Split upstream and subscribed lockfiles when bases differ. Never mix CentOS RPMs onto RHEL or AIPCC bases.
- Build from source or use approved dependencies (for example, AIPCC Python wheels). Avoid generic prefetch as much as possible. Work with ProdSec if you need an exception.
- Partner early for awkward binaries so multi-arch approved wheels exist before CI needs them.
- Patch upstream installers that download at postinstall time. The codeserver ripgrep pattern travels well.
- Automate lock renewal so "hermetic" does not mean "frozen forever".
- Budget CPU and memory for npm-heavy images. Document where GitHub Actions and Konflux diverge.
Where to go next
Here are recommendations for further reading, based on what you're trying to do:
- Understand the architecture: hermetic-guide.md
- Generate lockfiles and run prefetch: lockfile-generators README
- Copy a simple PipelineRun: runtime-minimal PR pipeline
- Study the complex npm case: codeserver README and codeserver PR pipeline
- Work with subscribed or AIPCC bases: subscribed-builds.md
- Validate Conforma locally: conforma.md
This implementation has been based on work in opendatahub-io/notebooks and the downstream red-hat-data-services/notebooks fork. Questions and improvements are welcome as issues and PRs on opendatahub-io/notebooks. The scripts and docs above are meant to be reused beyond our workbench set. If you are wrestling with air-gapped or Conforma requirements on Konflux or OpenShift AI images, we hope this gives you a head start.
