Serhei Makarov

Recent Posts

Installing debugging tools into a Red Hat OpenShift container with oc-inject

Installing debugging tools into a Red Hat OpenShift container with oc-inject

A previous article, Debugging applications within Red Hat OpenShift containers, gives an overview of tools for debugging applications within Red Hat OpenShift containers, and existing restrictions on their use. One of the restrictions discussed in that article was an inability to install debugging tool packages into an ordinary, unprivileged container once it was already instantiated. In such a container, debugging tool packages have to be included when the container image is built, because once the container is instantiated, using package installation commands requires elevated privileges that are not available to the ordinary container user.

However, there are important situations where it is desirable to install a debugging tool into an already-instantiated container. In particular, if the resolution of a problem requires access to the temporary state of a long-running containerized application, the usual method of adding debugging tools to the container by rebuilding the container image and restarting the application will destroy that temporary state.

To provide a way to add debugging tools to unprivileged containers, I developed a utility, called oc-inject, that can temporarily copy a debugging tool into a container. Instead of relying on package management or other privileged operations, oc-inject’s implementation is based on the existing and well-supported OpenShift operations oc rsync and oc exec, which do not require any elevated privileges.

This article describes the current capabilities of the oc-inject utility, which is available on GitHub or via a Fedora COPR repository. The oc-inject utility works on any Linux system that includes Python 3, the ldd utility, and the Red Hat OpenShift command-line tool oc.

Continue reading “Installing debugging tools into a Red Hat OpenShift container with oc-inject”

Debugging applications within Red Hat OpenShift containers

Debugging applications within Red Hat OpenShift containers

When debugging an application within a Red Hat OpenShift container, it is important to keep in mind that the Linux environment within the container is subject to various constraints. Because of these constraints, the full functionality of debugging tools might not be available:

  • An unprivileged OpenShift container is restricted from accessing kernel interfaces that are required by some low-level debugging tools.

Note: Almost all applications on OpenShift run in unprivileged containers. Unprivileged containers allow the use of standard debugging tools such as gdbserver or strace. Examples of debugging tools that cannot be used in unprivileged containers include perf, which requires access to the kernel’s perf_events interface, and SystemTap, which depends on the kernel’s module-loading functionality.

  • Debug information for system packages within OpenShift containers is not accessible. There is ongoing work (as part of the elfutils project) to develop a file server for debug information (debuginfod), which would make such access possible.
  • The set of packages in an OpenShift container is fixed ahead of time, when the corresponding container image is built. Once a container is running, no additional packages can be installed. A few debugging tools are preinstalled in commonly used container base images, but any other tools must be added when the container image build process is configured.

To successfully debug a containerized application, it is necessary to understand these constraints and how they determine which debugging tools can be used.

Continue reading “Debugging applications within Red Hat OpenShift containers”