Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Debugging applications within Red Hat OpenShift containers

January 9, 2020
Serhei Makarov
Related topics:
ContainersDevOpsKubernetes
Related products:
Developer ToolsRed Hat OpenShift LocalRed Hat OpenShiftRed Hat OpenShift Container Platform

Share:

    When debugging an application within a Red Hat OpenShift container, it is important to keep in mind that the Linux environment within the container is subject to various constraints. Because of these constraints, the full functionality of debugging tools might not be available:

    • An unprivileged OpenShift container is restricted from accessing kernel interfaces that are required by some low-level debugging tools.

    Note: Almost all applications on OpenShift run in unprivileged containers. Unprivileged containers allow the use of standard debugging tools such as gdbserver or strace. Examples of debugging tools that cannot be used in unprivileged containers include perf, which requires access to the kernel’s perf_events interface, and SystemTap, which depends on the kernel’s module-loading functionality.

    • Debug information for system packages within OpenShift containers is not accessible. There is ongoing work (as part of the elfutils project) to develop a file server for debug information (debuginfod), which would make such access possible.
    • The set of packages in an OpenShift container is fixed ahead of time, when the corresponding container image is built. Once a container is running, no additional packages can be installed. A few debugging tools are preinstalled in commonly used container base images, but any other tools must be added when the container image build process is configured.

    To successfully debug a containerized application, it is necessary to understand these constraints and how they determine which debugging tools can be used.

    Debugging tools preinstalled in container images

    Most container base images for the OpenShift platform include at least a minimal level of preinstalled debugging functionality. General-purpose base images such as the Red Hat Enterprise Linux (RHEL) Universal Base Image include the gdbserver component of the GDB debugger, while container base images intended for higher-level language runtimes usually include debugging functionality specific to those runtimes.

    The general-purpose Red Hat Enterprise Linux container base images include the gdbserver component of the commonly-used GDB debugger. Gdbserver can be used by configuring an interactive GDB session that runs outside the OpenShift cluster to pass remote commands to a gdbserver instance within a container. This gdbserver instance has direct access to the containerized application. In response to remote commands from the GDB session, gdbserver will collect information from the application and pass it back to the GDB session:

    Red Hat OpenShift containers

    The following debugging session illustrates how gdbserver can be attached to a program in a container based on the Red Hat Universal Base Image (UBI). The commands in this session should also work with containers based on earlier Red Hat Enterprise Linux container base images. This procedure was tested on an OpenShift 4.2.8 cluster managed with CodeReady Containers 1.2.0.

    We begin the session by creating a container:

    $ oc new-app --name='ubi8-example' nodeshift/ubi8-s2i-web-app:latest~https://github.com/lholmquist/react-web-app
    

    Obtain the name of the container with the oc get pods operation:

    $ oc get pods
    NAME                        READY   STATUS             RESTARTS   AGE
    ubi8-example-1-22wlk        1/1     Running            0          2m45s
    ubi8-example-1-build        0/1     Completed          0          5m20s
    ubi8-example-1-deploy       0/1     Completed          0          2m53s
    

    To verify that the container includes the gdbserver package, use the oc exec operation to execute an rpm command within the container:

    $ oc exec -it ubi8-example-1-22wlk -- rpm -qa | grep gdbserver
    gdb-gdbserver-8.2-6.el8.x86_64
    

    We need the process ID of the program we want to debug. To obtain it, execute a ps command within the container:

    $ oc exec -it ubi8-example-1-22wlk -- ps -ax
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:00 /bin/bash /usr/libexec/s2i/run
      9 ?        Sl     0:00 /usr/bin/node /usr/bin/npx serve -s /opt/app-root/output -l 
     48 pts/0    Rs+    0:00 ps -ax
    

    Next, launch a local GDB session and connect to gdbserver within the container:

    $ gdb
    (gdb) target extended-remote | oc exec -i ubi8-example-1-22wlk -- gdbserver --multi - 
    Remote debugging using | oc exec -i ubi8-example-1-22wlk -- gdbserver --multi -
    Remote debugging using stdio
    

    Here, GDB is instructed to connect to a gdbserver process in the container. This gdbserver process is started with an oc exec operation and configured to listen for commands on its standard input. The OpenShift infrastructure passes the standard input and standard output of the gdbserver process over the network to the local GDB session.

    Next, instruct GDB to attach to the main server process in the container, which is identified by the process ID we obtained earlier:

    (gdb) attach 9
    Attaching to process 9
    Attached; pid = 9
    Reading /usr/bin/node from remote target...
    ...
    0x00007fd1cb44228e in epoll_pwait () from target:/lib64/libc.so.6
    

    At this point, GDB is ready to debug the process.

    Restrictions on GDB functionality within OpenShift containers

    In a Red Hat OpenShift container, certain GDB functionality could be unavailable depending on how the container is configured.

    For example, GDB functions that depend on access to debug information will be unavailable. Although debug information packages are available in many Linux distributions, including Red Hat Enterprise Linux, these packages can only be included in a container image ahead of time, and cannot be installed in an unprivileged container once that container is already running.

    Nevertheless, there is a subset of GDB’s functions which can be used in the absence of debug information:

    • GDB can trigger breakpoints and tracepoints at function entry points.
    • GDB can use the mechanism of statically-defined tracing (SDT) markers in order to trigger breakpoints or tracepoints on various high-level events in the program.
    • GDB can generate a core file of the program’s memory. The core file can be saved and inspected outside of the OpenShift environment.
    • GDB can obtain stack traces from the program.

    The following example shows how the command thread apply all bt can be used to obtain a stack trace of each thread in a process:

        (gdb) attach 9
        (gdb) thread apply all bt
        
        Thread 11 (Thread 9.19):
        #0  0x00007fd1cb71747c in pthread_cond_wait@@GLIBC_2.3.2 ()
           from target:/lib64/libpthread.so.0
        #1  0x00005585ba200d7d in uv_cond_wait ()
        #2  0x00005585ba1eea22 in worker ()
        #3  0x00007fd1cb7112de in start_thread () from target:/lib64/libpthread.so.0
        #4  0x00007fd1cb442133 in clone () from target:/lib64/libc.so.6
        
        ...
        
        Thread 2 (Thread 9.10):
        #0  0x00007fd1cb44228e in epoll_pwait () from target:/lib64/libc.so.6
        #1  0x00005585ba203f7e in uv.io_poll ()
        #2  0x00005585ba1f3ea0 in uv_run ()
        #3  0x00005585ba10faa4 in node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start()::{lambda(void*)#1}::_FUN(void*) ()
        #4  0x00007fd1cb7112de in start_thread () from target:/lib64/libpthread.so.0
        #5  0x00007fd1cb442133 in clone () from target:/lib64/libc.so.6
        
        Thread 1 (Thread 9.9):
        #0  0x00007fd1cb44228e in epoll_pwait () from target:/lib64/libc.so.6
        #1  0x00005585ba203f7e in uv.io_poll ()
        #2  0x00005585ba1f3ea0 in uv_run ()
        #3  0x00005585ba0e3c7a in node::NodeMainInstance::Run() ()
        #4  0x00005585ba072eb4 in node::Start(int, char**) ()
        #5  0x00007fd1cb369873 in __libc_start_main () from target:/lib64/libc.so.6
        #6  0x00005585ba00f14e in _start ()
    

    Secondly, GDB functions that require stopping the application are incompatible with periodic health checks. If a container has been configured to run periodic health checks, the OpenShift infrastructure will periodically send a message to the containerized application to verify that it is running. If the application has been paused by GDB, it will fail to respond to the message and OpenShift will restart the container on the assumption that the application has crashed. This restart will interrupt the debugging session and reset the application state. In this situation, it is not feasible to pause the program for an extended period of time. For such situations, GDB supports background execution and dynamic tracing. GDB’s background execution commands, such as attach&, allow GDB to attach to a program without pausing it. GDB’s tracepoint facilities can collect data while the program continues running.

    Debugging tools for higher-level languages

    Container base images that include higher-level language runtimes might include corresponding debugging functionality. For example, Java container images include support for the Java remote debugging protocol, and Node.js container images include the Node.js inspector agent. An earlier article by Jeff Maury describes how to enable and use debugging features in container images for the Java runtime, and an article by Lucas Holmquist describes how to enable and use debugging features in container images for Node.js applications.

    Although many commonly used container base images include debugging facilities, there is no absolute guarantee that a container base image will include them. It is necessary to examine the contents of a container image in order to understand the included debugging facilities and determine whether additional packages need to be installed.

    Installing debugging tools at container image build time

    There are debugging tools that can be used within containers but are not preinstalled in container base images. Tools such as strace or Valgrind must be included in a container during the container image build process.

    In order to add a debugging tool to a container, the container image build process must be configured to perform additional package installation commands. Whether or not package installation is permitted during the image build process depends on the method being used to build the container image. OpenShift provides several methods of building container images. These methods are called build strategies. Currently, OpenShift supports the Dockerfile, Source-to-Image (S2I), Pipeline, and Custom build strategies. Not all build strategies allow package installation: Of the most commonly-used strategies, the Dockerfile strategy permits package installation but the S2I strategy does not, because an S2I build process builds the container image in an unprivileged environment. A build process within an unprivileged environment lacks the ability to invoke package installation commands.

    Debugging tool packages installed during the Dockerfile build process for a container image are included in that container image. For example, the following Dockerfile commands specify a container that includes the strace package:

    FROM registry.access.redhat.com/ubi8/ubi-init
    RUN yum -y install strace
    

    Debugging tools requiring privileged containers

    A debugging tool cannot operate within an OpenShift container if the container does not have the privileges to access the operating system functionality required by the tool. For example, the SystemTap default kernel module backend cannot operate in containers that lack the privileges to load kernel modules. Similarly, perf requires access to the kernel perf_events interface, and bcc requires access to BPF tracing functionality. These functions are not available within unprivileged OpenShift containers. In general, a debugging tool that requires elevated privileges can only be used within a privileged container created by a cluster administrator.

    In general, to determine whether a debugging tool can be used within an OpenShift container, it is necessary to investigate the set of operating system interfaces the tool relies on, and then to identify whether the OpenShift container has sufficient privileges to access these interfaces. Commonly used debugging tools such as gdbserver or strace are able to operate within unprivileged OpenShift containers because they rely on the ptrace() system call, which is available by default in unprivileged containers.

    Conclusion

    This article gave a summary of various options for debugging a containerized application on OpenShift. Currently, in order to troubleshoot a running OpenShift application, it is necessary to anticipate the set of debugging tools that might be required, and steps must be taken ahead of time to ensure that these debugging tools will be present within the container.

    Last updated: November 8, 2023

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue