Running containers as a privileged user in a Red Hat OpenShift cluster is a security issue. Installing a package, running nested containers, and other tasks that require sudoers privileges are usually not allowed in the cloud. When containers run with root privileges, they may compromise the host system, either intentionally or unintentionally.
This is problematic when running cloud development environments (CDEs) in an OpenShift cluster, such as Red Hat OpenShift Dev Spaces. Developer tools often require privileges that are beyond the default, restricted, privileges of an OpenShift container.
The challenge, therefore, is to run CDE in privileged containers without compromising cluster security. This may look contradictory, but as we will discuss in this article, the enhanced isolation provided by Kata containers makes it possible.
Note
This article explains a technique for running privileged commands in CDEs, including commands to install new packages. While installing a new tool can be useful when experimenting with it, we discourage updating CDEs in this manner. Containers should be considered immutables. Any newly installed package will disappear after restarting the CDE. For the recommended way to update an OpenShift Dev Spaces CDE, please refer to this blog.
Kata containers to the rescue
Kata containers provide pod sandboxing by leveraging virtualization. Each pod runs in a VM, isolating its containers from the host system. This isolation allows running privileged pods while still protecting the host. Running as root inside the container is no longer a security issue.
The OpenShift sandboxed containers solution brings Kata containers support to OpenShift.
The diagram in Figure 1 compares a standard pod to an OpenShift sandboxed container pod. Note the additional green VM layer for isolation.
You can read more about OpenShift sandboxed containers in this blog post.
A policy to run privileged and secured containers
Service accounts, along with roles and roleBindings, define the actions a user or application can perform within the OpenShift cluster. To enable the creation of privileged containers, we define a privileged role and assign it to a specific service account. This service account will have the necessary permissions to create privileged containers.
Further, to ensure these privileged containers don't compromise the host, we can implement a Kyverno policy that enforces the use of sandboxed (Kata) containers for any pods created by this privileged service account. This ensures that any privileged container cannot impact the host.
Configuring OpenShift sandboxed containers
Follow the official product guide to deploy OpenShift sandboxed containers on bare metal worker nodes.
Red Hat OpenShift Dev Spaces runs cloud development environments (CDEs). The default IDE running in the pod is Visual Studio Code. By default, the CDEs are unprivileged, so dnf install and podman run won’t work. But if you have configured the cluster like the Kata containers and privileged service account in the previous section (limited by a Kyverno policy for each developer namespace), then we can configure Dev Spaces to run the CDE as privileged Kata containers, where dnf install and podman run will work.
Figure 2 shows that, when Dev Spaces uses Kata containers, privileged commands such as podman run -ti --rm hello-world execute successfully.
You can find the step-by-step instructions on how to set up a cluster, OpenShift sandboxed containers operator, OpenShift Dev Spaces operator, and Kyverno in the Red Hat Developer repository.
Limits and alternatives
Running Kata containers requires a Kubernetes cluster with bare-metal worker nodes. We did our tests using an OpenShift cluster with AWS m5.metal worker nodes. The OpenShift cluster bot provisions such a cluster with the launch 4.20 metal,ovn command.
Another approach to run privileged commands in a CDE is to enable user namespaces in containers so the container's root user doesn’t match the node's root user. Recently, we added this feature to OpenShift 4.20, which is extensively covered in this article.
Final thoughts
In this article, you learned about a technique for running privileged commands within cloud development environments (CDEs) in OpenShift Dev Spaces, addressing the common security issue of running containers as a privileged user in an OpenShift cluster. The solution is using Kata containers (provided by OpenShift sandboxed containers), which use virtualization to run each pod in a separate VM. This isolation protects the host system, allowing privileged containers to run without compromising cluster security. For more information, you can visit our GitHub repository.
