Using sidecars to analyze and debug network traffic in OpenShift and Kubernetes pods

In the world of distributed computing, containers, and microservices, a lot of the interactions and communication between services is done via RESTful APIs. While developing these APIs and interactions between services, I often have the need to debug the communication between services, especially when things don't seem to work as expected.

Before the world of containers, I would simply deploy my services on my local machine, start up Wireshark, execute my tests, and analyze the HTTP communication between my services. This for me has always been an easy and effective way to quickly analyze communication problems in my software. However, this method of debugging does not work well in a containerized world.

First of all, the containers most likely run on an internal container platform network that is not directly accessible by your machine. A second problem is that, in compliance with container design best practices, containers contain only the minimal set of applications and libraries needed to execute their task. This means that a tool like tcpdump is usually not available in a container. This makes debugging and analyzing network traffic between containers and, thus, debugging of inter-microservice communication a bit harder than in the non-containerized world. This article shows one solution.

Sidecars to the rescue

In the last couple of months, I've tried various approaches to overcome this problem, which resulted in the approach that I'll outline in this article. It's an easy way to capture network traffic data between Kubernetes/OpenShift pods, allowing developers to better analyze and debug communication problems in containerized applications and to solve issues quicker and more effectively.

We will be using tcpdump to capture a so-called, PCAP (packet capture) file that will contain the pod's network traffic. This PCAP file can then be loaded in a tool like Wireshark to analyze the traffic and, in this case, the RESTful communication of a service running in a pod. In this article, I'll be using the KIE Server (execution server) of the Red Hat Process Automation Manager product as an example, but this approach should work with any kind of containerized application.

The first problem to overcome is the availability of the tcpdump command in the Kubernetes pod. The KIE Server container image does not have tcpdump installed. Second, the container does not provide the utilities to install tcpdump from the Red Hat repositories. To overcome this problem, we use the concept of a "sidecar container."

A sidecar container is a container that is running in the same pod as the actual service/application and is able to provide additional functionality to the service/application. An example of a sidecar container is Istio's Envoy sidecar, which enables a pod to become part of a service mesh. In this example, we will be deploying a sidecar container that provides the tcpdump utility. Because multiple containers in a pod share the same network layer, we can use the sidecar to capture network traffic to and from the KIE Server.

Deploying the sidecar

In this example, I've deployed the Red Hat Process Automation Manager 7 Mortgage Demo, which will create two pods in my OpenShift namespace. One pod runs the Business Central workbench, and the other pod is the pod for the execution server. Communication between the two components is done via REST, which is the traffic we're going to capture.

Our aim is to capture the network traffic on the KIE Server pod, so we can analyze the RESTful commands sent by the Business Central workbench to the KIE Server. To do this, we first need to attach a sidecar to the KIE Server pod:

1. In the Overview screen, click the name of the pod that you want to analyze. This will open the Deployment Config (DC) screen.
2. In the Deployment Config screen, in the upper right corner, click "Actions -> Edit YAML." This will open the DC's YAML configuration.

3. Scroll down until you see the word "containers." We're going to add an additional container, our sidecar with tcpdump installed, to our pod. Add the following YAML snippet directly under the "containers" definition:

- name: tcpdump
   image: corfr/tcpdump
   command:
     - /bin/sleep
     - infinity

4. Save the configuration. This will deploy a new pod that will now consist of two containers: one container with the KIE Server and the other container that contains our tcpdump tool and which will keep running indefinitely.

Note that running a "sleep" command inside a container to keep it running is a bit of a "hack" and definitely not a good practice for containers running in production. However, after several failed approaches, which included running tcpdump as the container command, this approach made me achieve my goal in a controllable way.

Capturing and analyzing traffic

With the sidecar deployed and running, we can now start capturing data. One of the approaches I tried was to use the oc rsh command to remotely execute the tcpdump command inside the sidecar, stream the output to a FIFO file, and pipe the data directly into Wireshark. This approach failed for various reasons. One of the problems was that tcpdump sends info messages to stderr, but these messages will be received over SSH in the same stream as stdout, corrupting the data going into Wireshark.

The approach I ended up using is to log in to the sidecar container and, within the sidecar, run the tcpdump command to create a PCAP file. When you've captured enough data, you can stop the capturing process and copy the PCAP file to the machine on which you want to do the network traffic analysis with Wireshark. This is done as follows:

1. On your development machine, with your oc client connected to your OpenShift instance and with the correct project (namespace) active, run the oc get pods command to list your pods:

2. Log in to the tcpdump container of our KIE Server pod with the following command: oc rsh -c tcpdump rhpam7-mortgage-kieserver-2-zcpsn.
3. In the tcpdump container, run this command to start the network traffic capturing process: tcpdump -s 0 -n -w /tmp/kieserver.pcap.
4. Run the tests that create the network traffic you want to analyze. In this example, I'll be starting a Business Process from the Business Central workbench, which will send a RESTful request to the KIE Server.
5. When you've captured enough data, finish the capturing process by using Ctrl+C in the tcpdump container.
6. Go back to your local machine. Copy the PCAP file from the pod to your local machine: oc cp -c tcpdump rhpam7-mortgage-kieserver-2-zcpsn:tmp/kieserver.pcap kieserver.pcap.
7. Open the PCAP file with Wireshark and analyze your network traffic. In this example, I'm analyzing my HTTP POST method that creates a new instance of our Mortgage process:

Conclusion

Analyzing network traffic between pods in a container environment like Kubernetes and/or OpenShift can be a bit harder than in non-container environments. However, the concept of sidecar containers gives developers an easy tool to attach containers, with the needed development tools and utilities, to a microservice pod. This prevents developers from having to install these kinds of debugging tools in the application container image itself, keeping the container light and clean. Using OpenShift tools like oc rsh and oc cp, I have shown how you can easily capture network traffic data from a pod and bring the data to a development machine for analysis.

If you're interested in learning more about using Istio Egress when your microservice needs to talk to another service that is outside of your OpenShift/Kubernetes/pods environment, see Istio Egress: Exit Through the Gift Shop.

Last updated: September 3, 2019