Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Using sidecars to analyze and debug network traffic in OpenShift and Kubernetes pods

 

February 27, 2019
Duncan Doyle
Related topics:
Kubernetes
Related products:
Red Hat OpenShift

Share:

    In the world of distributed computing, containers, and microservices, a lot of the interactions and communication between services is done via RESTful APIs. While developing these APIs and interactions between services, I often have the need to debug the communication between services, especially when things don't seem to work as expected.

    Before the world of containers, I would simply deploy my services on my local machine, start up Wireshark, execute my tests, and analyze the HTTP communication between my services. This for me has always been an easy and effective way to quickly analyze communication problems in my software. However, this method of debugging does not work well in a containerized world.

    First of all, the containers most likely run on an internal container platform network that is not directly accessible by your machine. A second problem is that, in compliance with container design best practices, containers contain only the minimal set of applications and libraries needed to execute their task. This means that a tool like tcpdump is usually not available in a container. This makes debugging and analyzing network traffic between containers and, thus, debugging of inter-microservice communication a bit harder than in the non-containerized world. This article shows one solution.

    Sidecars to the rescue

    In the last couple of months, I've tried various approaches to overcome this problem, which resulted in the approach that I'll outline in this article. It's an easy way to capture network traffic data between Kubernetes/OpenShift pods, allowing developers to better analyze and debug communication problems in containerized applications and to solve issues quicker and more effectively.

    We will be using tcpdump to capture a so-called, PCAP (packet capture) file that will contain the pod's network traffic. This PCAP file can then be loaded in a tool like Wireshark to analyze the traffic and, in this case, the RESTful communication of a service running in a pod. In this article, I'll be using the KIE Server (execution server) of the Red Hat Process Automation Manager product as an example, but this approach should work with any kind of containerized application.

    The first problem to overcome is the availability of the tcpdump command in the Kubernetes pod. The KIE Server container image does not have tcpdump installed. Second, the container does not provide the utilities to install tcpdump from the Red Hat repositories. To overcome this problem, we use the concept of a "sidecar container."

    A sidecar container is a container that is running in the same pod as the actual service/application and is able to provide additional functionality to the service/application. An example of a sidecar container is Istio's Envoy sidecar, which enables a pod to become part of a service mesh. In this example, we will be deploying a sidecar container that provides the tcpdump utility. Because multiple containers in a pod share the same network layer, we can use the sidecar to capture network traffic to and from the KIE Server.

    Deploying the sidecar

    In this example, I've deployed the Red Hat Process Automation Manager 7 Mortgage Demo, which will create two pods in my OpenShift namespace. One pod runs the Business Central workbench, and the other pod is the pod for the execution server. Communication between the two components is done via REST, which is the traffic we're going to capture.

    Red Hat Process Automation Manager 7 mortgage demo deployed

    Our aim is to capture the network traffic on the KIE Server pod, so we can analyze the RESTful commands sent by the Business Central workbench to the KIE Server. To do this, we first need to attach a sidecar to the KIE Server pod:

    1. In the Overview screen, click the name of the pod that you want to analyze. This will open the Deployment Config (DC) screen.
    2. In the Deployment Config screen, in the upper right corner, click "Actions -> Edit YAML." This will open the DC's YAML configuration.

    Deployment Config screen

    1. Scroll down until you see the word "containers." We're going to add an additional container, our sidecar with tcpdump installed, to our pod. Add the following YAML snippet directly under the "containers" definition:
    - name: tcpdump
       image: corfr/tcpdump
       command:
         - /bin/sleep
         - infinity

    Adding a YAML snippet

    1. Save the configuration. This will deploy a new pod that will now consist of two containers: one container with the KIE Server and the other container that contains our tcpdump tool and which will keep running indefinitely.

    Note that running a "sleep" command inside a container to keep it running is a bit of a "hack" and definitely not a good practice for containers running in production. However, after several failed approaches, which included running tcpdump as the container command, this approach made me achieve my goal in a controllable way.

    Capturing and analyzing traffic

    With the sidecar deployed and running, we can now start capturing data. One of the approaches I tried was to use the oc rsh command to remotely execute the tcpdump command inside the sidecar, stream the output to a FIFO file, and pipe the data directly into Wireshark. This approach failed for various reasons. One of the problems was that tcpdump sends info messages to stderr, but these messages will be received over SSH in the same stream as stdout, corrupting the data going into Wireshark.

    The approach I ended up using is to log in to the sidecar container and, within the sidecar, run the tcpdump command to create a PCAP file. When you've captured enough data, you can stop the capturing process and copy the PCAP file to the machine on which you want to do the network traffic analysis with Wireshark. This is done as follows:

    1. On your development machine, with your oc client connected to your OpenShift instance and with the correct project (namespace) active, run the oc get pods command to list your pods:

    Listing your pods

    1. Log in to the tcpdump container of our KIE Server pod with the following command: oc rsh -c tcpdump rhpam7-mortgage-kieserver-2-zcpsn.
    2. In the tcpdump container, run this command to start the network traffic capturing process: tcpdump -s 0 -n -w /tmp/kieserver.pcap.
    3. Run the tests that create the network traffic you want to analyze. In this example, I'll be starting a Business Process from the Business Central workbench, which will send a RESTful request to the KIE Server.
    4. When you've captured enough data, finish the capturing process by using Ctrl+C in the tcpdump container.
    5. Go back to your local machine. Copy the PCAP file from the pod to your local machine: oc cp -c tcpdump rhpam7-mortgage-kieserver-2-zcpsn:tmp/kieserver.pcap kieserver.pcap.
    6. Open the PCAP file with Wireshark and analyze your network traffic. In this example, I'm analyzing my HTTP POST method that creates a new instance of our Mortgage process:

    Analyzing the network traffic

    Conclusion

    Analyzing network traffic between pods in a container environment like Kubernetes and/or OpenShift can be a bit harder than in non-container environments. However, the concept of sidecar containers gives developers an easy tool to attach containers, with the needed development tools and utilities, to a microservice pod. This prevents developers from having to install these kinds of debugging tools in the application container image itself, keeping the container light and clean. Using OpenShift tools like oc rsh and oc cp, I have shown how you can easily capture network traffic data from a pod and bring the data to a development machine for analysis.

    If you're interested in learning more about using Istio Egress when your microservice needs to talk to another service that is outside of your OpenShift/Kubernetes/pods environment, see Istio Egress: Exit Through the Gift Shop.

    Duncan Doyle

    About the author

    Duncan Doyle is the Technical Marketing Manager for the Red Hat Decision Manager and Red Hat Process Automation Manager platforms. With a background in Red Hat Consulting and Services, Duncan has worked extensively with large Red Hat customers to build advanced, open-source business rules and business process management solutions.

    He has a strong background in technologies and concepts like Service Oriented Architecture, Continuous Integration and Continuous Delivery, rules engines, and BPM platforms, and he is a subject matter expert (SME) on multiple Red Hat Middleware technologies, including, but not limited to, Red Hat JBoss Enterprise Application Platform, HornetQ, Red Hat Fuse, Red Hat Data Grid, Red Hat Decision Manager, and Red Hat Process Automation Manager. When he’s not working on open-source solutions and technology, he is building Lego with his son and daughter or jamming some 90’s rock music on his Fender Stratocaster.

    Last updated: March 27, 2023

    Recent Posts

    • AI meets containers: My first step into Podman AI Lab

    • Live migrating VMs with OpenShift Virtualization

    • Storage considerations for OpenShift Virtualization

    • Upgrade from OpenShift Service Mesh 2.6 to 3.0 with Kiali

    • EE Builder with Ansible Automation Platform on OpenShift

    What’s up next?

     

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue