Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Using OpenShift 4's inspect for middleware troubleshooting

May 16, 2024
Francisco De Melo Junior Alexander Barbosa Ayala Will Russell
Related topics:
JavaKubernetes
Related products:
Red Hat OpenShiftRed Hat OpenShift Container Platform

Share:

    Given a problem with an application in Red Hat OpenShift Container Platform (RHOCP 4) upon opening a case with the support team, our support team might ask for a namespace’s inspect file. The inspect file is a very useful tool for troubleshooting applications and systems deployed in RHOCP 4 and can be extremely useful in a series of situations.

    It also avoids the unnecessarily separated collection of data (Pod Logs, route YAML, BuildConfig YAMLs) given it brings everything that is namespace bounded—except custom resources—directly to one goal. 

    This article covers what the inspect file is, how to collect it, how to use it for troubleshooting, and how it compares to the must-gather file (spoiler: it is not the same thing). This will help clarify why we usually ask for this file during the troubleshooting procedures in middleware products, such as Red Hat JBoss Enterprise Application Platform (JBoss EAP), Red Hat Data Grid, Red Hat JBoss Web Server, and Spring Boot applications (which Red Hat supports for the Spring Boot 3+ community).

    inspect

    The inspect is a collection of a namespace’s specific files, also known as the project dump, that collects a set of most common namespace-bounded files: DC, pod YAMLs, routes, services, and so much more.

    It can be used to discover problems such as:

    • On the deployment of a specific pod
    • A specific pod issue
    • An issue that happens only on that namespace
    • An image pull issue happening currently (given a problem with the BuildConfig)
    • A BuildConfig issue or misconfiguration

    How to collect the inspect file

    The collection of the inspect is pretty straightforward, but requires admin rights on the cluster (which require adm permissions):

    $ oc adm inspect namespace namespace_name

    Content

    The inspect file will contain several namespace-bounded resources, including YAML and pod YAMLs/log. See the complete output of its content below:

    ├── inspect-logs
    │   └── inspect.local.onelocal
    │      ├── event-filter.html
    │      ├── namespaces
    │      │   └── datagrid <------------------------------------ datagrid namespace
    │      │       ├── apps
    │      │       │   ├── daemonsets.yaml
    │      │       │   ├── deployments.yaml
    │      │       │   ├── replicasets.yaml
    │      │       │   └── statefulsets.yaml
    │      │       ├── apps.openshift.io
    │      │       │   └── deploymentconfigs.yaml
    │      │       ├── autoscaling
    │      │       │   └── horizontalpodautoscalers.yaml
    │      │       ├── batch
    │      │       │   ├── cronjobs.yaml
    │      │       │   └── jobs.yaml
    │      │       ├── build.openshift.io
    │      │       │   ├── buildconfigs.yaml <------------------ buildconfig
    │      │       │   └── builds.yaml <------------------------ build
    │      │       ├── core
    │      │       │   ├── configmaps.yaml <--------------------- config maps
    │      │       │   ├── endpoints.yaml
    │      │       │   ├── events.yaml <------------------------- events
    │      │       │   ├── persistentvolumeclaims.yaml
    │      │       │   ├── pods.yaml <--------------------------- pod yamls
    │      │       │   ├── replicationcontrollers.yaml
    │      │       │   ├── secrets.yaml       <---------------------- secrets
    │      │       │   └── services.yaml      <--------------------- services
    │      │       ├── image.openshift.io
    │      │       │   └── imagestreams.yaml
    │      │       ├── pods
    │      │       │   ├── infinispan-operator-controller-manager        <------------operator logs
    │      │       │   │   ├── infinispan-operator-controller-manager.yaml
    │      │       │   │   └── manager
    │      │       │   │       └── manager
    │      │       │   │           └── logs
    │      │       │   │               ├── current.log
    │      │       │   │               └── previous.log
    │      │       │   ├── prometheus-prometheus <------------------------- prometheus
    │      │       │   │   ├── config-reloader
    │      │       │   │   │   └── config-reloader
    │      │       │   │   │       └── logs
    │      │       │   │   │           ├── current.log
    │      │       │   │   │           ├── previous.insecure.log
    │      │       │   │   │           └── previous.log
    │      │       │   │   ├── prometheus
    │      │       │   │   │   └── prometheus
    │      │       │   │   │       └── logs
    │      │       │   │   │           ├── current.log
    │      │       │   │   │           └── previous.log
    │      │       │   │   └── prometheus-prometheus.yaml
    │      │       │   └── datagrid-pod
    │      │       │       ├── infinispan
    │      │       │       │   └── infinispan
    │      │       │       │       └── logs
    │      │       │       │           ├── current.log <------------------------- pod logs
    │      │       │       │           ├── previous.insecure.log
    │      │       │       │           └── previous.log
    │      │       │       └── datagrid-cluster.yaml
    │      │       ├── policy
    │      │       │   └── poddisruptionbudgets.yaml
    │      │       ├── datagrid.yaml
    │      │       └── route.openshift.io
    │      │           └── routes.yaml                  <------------------------- routes yaml
    │      └── timestamp
    

    In the inspect directory, it is common to look for more relevant data/configuration available there. Here are some of the directories and file locations more frequently searched into a generated inspect:

    • Deployment pod logs: Go to the deployment directory name. In this case, the deployment name is rhdg-cluster:

      ├── inspect-logs
              │   └── inspect.local.onelocal
              │      ├── event-filter.html
              │      ├── namespaces
              │      │   └── datagrid
              …
              │      │       │   └── datagrid
              │      │       │       ├── infinispan
              │      │       │       │   └── infinispan
              │      │       │       │       └── logs
              │      │       │       │           ├── current.log 
              │      │       │       │           ├── previous.insecure.log
              │      │       │       │           └── previous.log
              │      │       │       └── datagrid.yaml
    • Deployment and StatefulSets configuration: Depending on the deployment process and requirements, a deployment could contain both a deployment and a StatefulSet. Both are located in the same directory:

      ├── inspect-logs
              │   └── inspect.local.onelocal
              │      ├── event-filter.html
              │      ├── namespaces
              │      │   └── rhdg
              …
              │      │       ├── apps
              │      │       │   ├── daemonsets.yaml
              │      │       │   ├── deployments.yaml
              │      │       │   ├── replicasets.yaml
              │      │       │   └── statefulsets.yaml
    • Deployment core details: Each deployment could contain a set of ConfigMaps, services, secrets, and so on, that are part of the fine-grained details of the deployment definition. Those configurations are available in the core directory:

      ├── inspect-logs
              │   └── inspect.local.onelocal
              │      ├── event-filter.html
              │      ├── namespaces
              │      │   └── datagrid
              …
              │      │       ├── core
              │      │       │   ├── configmaps.yaml 
              │      │       │   ├── endpoints.yaml
              │      │       │   ├── events.yaml
              │      │       │   ├── persistentvolumeclaims.yaml
              │      │       │   ├── pods.yaml
              │      │       │   ├── replicationcontrollers.yaml
              │      │       │   ├── secrets.yaml     
              │      │       │   └── services.yaml     
    • Routes: The created routes have their own file in a separate directory. Route details are in the route.openshift.io directory:

      ├── inspect-logs
              │   └── inspect.local.onelocal
              │      ├── event-filter.html
              │      ├── namespaces
              │      │   └── rhdg
              …
              ​​│      │       └── route.openshift.io
              │      │           └── routes.yaml 

    Tools for debugging

    For reviewing those files inside the inspect, there is no need to manually read each one of them or even grep the data because there are tools that do that: you can use OMG and OMC tools to parse both the inspect and the must-gather.

    To set for usage:

    $ omg use inspect.local.file
    .Using:  /home/path/inspect.local.file

    To get the pods:

    $ omg get pod -n a-namespace-prod  
    NAME                             READY  STATUS   RESTARTS  AGE
    apod-id    1/1    Running  0         4h2m
    apod2-id     1/1    Running  0         4h2m

    BuildConfig

    The inspect becomes a powerful tool to debug and troubleshoot BuildConfig, because each build step will be on the inspect, including the image streams:

    • BuildConfig
    • Build
    • Pod build logs
    • ImageStreams
    • Deployment pods
    • Application pods

    For investigations on build, get the BuildConfig YAML and verify the specific details on the build, like a Dockerfile. From the build YAML and the build Pod logs, one can see each step of the build to determine any specific step with issues.

    Example troubleshooting

    The inspect file can be used for numerous issues and provide hints even for RHOCP cluster ones. Below are two examples—a service spec issue and a build configuration issue.

    JBoss EAP pods do not cluster , or Service does not have the PublishNotReady spec

    Problem: When deploying JBoss EAP 7 and Red Hat Data Grid, the pods must form a proper cluster. To form the cluster those products use JGroups library, which is a Java library for one-to-one and one-to-many communication that forms the clusters. 

    Cause: In OpenShift Container Platform 4, a service is used to establish the connection between pods; in this case, JGroups will rely on a ping-service that binds the pods together. However, they must be ready immediately after creation—this is a requirement by JGroups, which is done via the spec.publishNotReadyAddresses property. This is a requirement from JGroups itself. 

    In summary, this service setting will allow the DNS resolution of that pod even when the pods are not ready (as in green-lighted by the readiness probe).

    Steps to troubleshoot and fix:

    1. Collect the inspect.
    2. When verifying the services YAMLs, it is possible to verify the service YAMLs, then correlate which one is associated with the JBoss EAP pods specifically, and then verify if that service has the correct spec.publishNotReadyAddresses. 
    3. To fix the problem, fix the service's spec section accordingly.

    BuildConfig brings different behaviors in two different images, even with the same tag

    Building images in RHOCP 4 can be very practical. However, if a problem arises, the inspect file can be a useful tool for discovery.

    Problem: When the built image using OpenJDK 17 image, the newest tag (as in the latest images in the registry) had a different behavior than older image tags.

    Cause: This was a change on the OpenJDK 17 images specifically rooted in the behavior change on JAVA_OPTS vs JAVA_OPTS_APPEND environment variables in the image, and this was an expected behavior. More information here.

    Steps to troubleshoot and fix: Through the collection of the inspect file, and then comparing the images deployed (ImageStream) and the BuildConfig and the build logs, we see that the tag difference (and digest image ID) on the OpenJDK image made the new image behave differently.

    Complementary to that, see the following solutions:

    • For generic customization parameters for Maven, see Using Maven parameters on BuildConfig in RHOCP 4.
    • For troubleshooting steps for JBoss EAP 7's BuildConfig, see Troubleshooting EAP 7 local BuildConfig in RHOCP 4.
    • For customizing template BuildConfig, see Customizing EAP 7 Template BuildConfig deployment in RHOCP 4.

    Spring Boot application deployment is unresponsive or slow

    Spring Boot is the Tomcat Catalina engine, so it is a Java application.

    Problem: When deployed in RHOCP 4 via deployment.yaml, for example, its container's CPU settings must be adequately set to avoid slowness or kernel starvation.

    Cause: Spring Boot will bring by default the threads set at 200 threads instead of a value deduced/calculated from the CPU settings.

    Steps to troubleshoot and fix: Collecting the inspect file from the pod YAML, we review the CPU containers settings and verify if that's enough. Although the inspect will only state how much it set (not the required for a certain specific application), in some cases—and with enough know-how on the matter/application—it is possible to deduce if the amount is enough to be set.

    Inspect versus must-gather (mg)

    Comparing the inspect bundle versus the must-gather ($ oc adm must-gather), the namespace’s inspect is not the same as the must-gather; it will contain only specific namespace details and not cluster details. must-gather is a much more complex collection of files, from several namespaces including—and mostly focused—on Openshift cluster system namespaces.

    In summary, every must-gather collects inspect + cluster data, must-gather is a collection of specified scripts that collect data from the cluster, which includes output from multiple inspect associated with those namespaces.

    The inspect specifically only collects namespace data (or data related to the collected objects).

     

    Namespace's Inspect structure
    Namespace's Inspect structure
    Figure 1: The namespace's Inspect concerns only one namespace data

    In this matter, the namespace's inspect file will only namespace information, whereas the must-gather has a collective of information from the cluster.

    must-gather:

    • Types of problems: Cluster-wide.
    • How many namespaces? Several, including system ones.
    • Does it have CRs? From some resources, yes.
    • The $ oc adm must-gather command is translated for oc run collect-pod --image=mustgatherimage -- sh gather, which creates a simple/single pod that will collect the data and at the end, it will run oc cp <pod>:/mustgather/dir ./ For specific details see openshift/must-gather/collection-scripts/gather. 

    inspect:

    • Types of problems: Namespace bounded.
    • How many namespaces? One—the one the user selects.
    • Does it have CRs? No, only default RHOCP and Kubernetes files.

    Conclusion

    In this article, we discussed the inspect file—how to get it, what's inside of it, and how to use it for troubleshooting, with two real-life examples. Finally, we compared the inspect bundle with the must-gather, which is another handy tool for cluster-wide issues in Red Hat OpenShift Container Platform 4.

    Additional resources

    Now you know this tool and how it can be useful for support cases, as detailed in the solution Using inspect for DG 8 troubleshooting.

    For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.

    Related Posts

    • What's new for developers in Red Hat OpenShift 4.15

    • How to update to newer Red Hat OpenShift 4 releases

    • Securely connect Quarkus and Red Hat Data Grid on Red Hat OpenShift

    • Extracting information from Python source code

    • Troubleshooting application performance with Red Hat OpenShift metrics, Part 1: Requirements

    • Troubleshooting "no healthy upstream" errors in Istio service mesh

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    What’s up next?

    Read Operating OpenShift, a practical guide to running and operating OpenShift clusters more efficiently using a site reliability engineering (SRE) approach. Learn best practices and tools that can help reduce the effort of deploying a Kubernetes platform.

    Get the e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue