Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Troubleshooting Red Hat OpenShift applications with throwaway containers

August 22, 2019
Fernando Lozano
Related topics:
LinuxContainersKubernetes
Related products:
Red Hat Enterprise LinuxRed Hat OpenShift

Share:

    Imagine this scenario: Your cool microservice works fine from your local machine but fails when deployed into your Red Hat OpenShift cluster. You cannot see anything wrong with the code or anything wrong in your services, configuration maps, secrets, and other resources. But, you know something is not right. How do you look at things from the same perspective as your containerized application? How do you compare the runtime environment from your local application with the one from your container?

    If you performed your due diligence, you wrote unit tests. There are no hard-coded configurations or hidden assumptions about the runtime environment. The cause should be related to the configuration your application receives inside OpenShift. Is it time to run your app under a step-by-step debugger or add tons of logging statements to your code?

    We'll show how two features of the OpenShift command-line client can help: the oc run and oc debug commands.

    Starting throwaway containers on Red Hat OpenShift

    Most developers think about containers and OpenShift only for running long-lived applications. You create deployment configurations, stateful sets, or cron jobs that stay alive forever, creating and re-creating pods as required. Your application is always on, or at least on at fixed intervals.

    The oc run command runs containers that perform a single task and then die. It creates unmanaged containers, that OpenShift does not replace when they die.

    I once had an application that talked to a legacy database outside of my OpenShift cluster. The application was able to access the database from my local machine, but not from OpenShift. I needed the ability to test access to the database from inside OpenShift. This way I could find out whether I got the correct environment variables. I would also see whether the container could resolve the database’s host name. Maybe there was a firewall blocking access to the database?

    This is a perfect scenario for the oc run command. Just start a pod running the database container image. From that pod, you can use the database client and OS commands to troubleshoot configuration network connectivity. After a few quick tests, you don't need the pod anymore.

    $ oc run -it test --rm --restart Never \
    --image registry.access.redhat.com/rhscl/mysql-57-rhel7 bash
    

    The previous command gives me an interactive (-it) Bash prompt on a pod named test. OpenShift never restarts this pod (--restart Never) and removes it when terminated (--rm).

    The MySQL database image from Red Hat (rhscl/mysql-57-rhel7) provides the MySQL client and a few other useful commands, such as dig and host. With this, I can check that I can resolve the server host name, connect to the database, and verify my access credentials.

    Starting throwaway containers for management clients

    I could start the MySQL client, or any other command available from that container image, directly from the oc run command. For example:

    $ oc run -it test --rm --restart Never \
    --image registry.access.redhat.com/rhscl/mysql-57-rhel7 \
    -- mysql -umydbuser -pmydbpassword -hmyserver.domain.example.com mydb

    Note the use of a double dash (--) to prevent the oc run command from interpreting the command options intended for the MySQL client. In the previous command, there is no --mysql option; there is a space between -- and mysql.

    As another example, I could start the same throwaway container from the first example, then use another terminal to copy a SQL script into the container using the oc cp command. Then I can run the SQL script using the throwaway container shell.

    Because the MySQL client can take SQL scripts from the standard input, I could just add input redirection to the second example and be done. I just populated a test database. What about doing this from a shell script or an Ansible playbook, while I do not write that fancy operator that would deploy and initialize the database for my application?

    Thanks to the oc run command, I can use administration clients embedded into many container images, for example, the CLI administration tools for JBoss EAP, AMQ, and single-sign on. I do not need to install any of them on my local machine. Cool, isn’t it?

    Cloning a deployment to a debug container

    I could add more command-line options to the oc run command and replicate all the settings of an existing deployment: environment variables, resource limits, and so on. If my intent were to replicate the runtime environment of my application, this would be too much work and be prone to errors.

    However, this would be a scenario for the oc debug command. It creates a new pod that is a carbon copy of an existing pod. If your pod does not start for whatever reason, you can create the copy from its deployment configuration.

    Suppose that I created my application using oc new-app and named it myapp. To create a debug pod from its deployment configuration, I would use the following command:

    $ oc debug -t dc/myapp

    I get a Bash shell running under the same restraints as my application: uid, SElinux context, environment variables, and the same container image.

    If I suspect that some of these restraints may be causing a failure, I can selectively override them using options from the oc debug command. For example, adding the --as-root option to the previous example gives me a root prompt inside the pod, but only if my OpenShift user has access to a security context constraint that allows me to do so.

    The debug pod runs with health probes disabled. I can start my application manually to check whether the health probes are incorrect and forcing my pod to terminate. I could add options to the oc debug command to enable health probes, disable init containers, disable sidecar containers, change labels that affect pod scheduling, and thus find which, if any, of the deployment settings are not correct for my application.

    Starting throwaway containers with RHEL tools container images

    As with the oc run command, your actions using the oc debug command are limited by what is included with your application container image. Fortunately, you can override the container image in your debug container. Good candidates are the rhel7/tools and the rhel8/support-tools container images from Red Hat.

    $ oc debug -t dc/myapp \
    --image registry.access.redhat.com/rhel7/rhel-tools

    These images provide access to standard RHEL troubleshooting commands that would not be included in most application images, for example, the ping and dig commands.

    You'll need to download the rhel8/support-tools container image from the Red Hat terms-based registry (redhat.registry.io). Access to the terms-based registry requires a pull secret. Follow the instructions from Red Hat Enterprise Linux Support Tools if needed.

    Conclusion

    You do not need a local container engine to run throwaway containers that perform troubleshooting and one-time tasks. You can run these containers quickly and easily on Red Hat OpenShift using the oc run and oc debug commands. And, your OpenShift cluster, if it is not a Minishift instance, is probably quicker to download container images and likely has more storage space and better bandwidth than your local workstation.

    Last updated: September 3, 2019

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue