Debugging Memory Issues with Open vSwitch DPDK


This article is about debugging out-of-memory issues with Open vSwitch with the Data Plane Development Kit (OvS-DPDK). It explains the situations in which you can run out of memory when using OvS-DPDK and it shows the log entries that are produced in those circumstances. It also shows some other log entries and commands for further debugging.

When you finish reading this article, you will be able to identify that you have an out-of-memory issue and you’ll know how to fix it. Spoiler: Usually having some more memory on the relevant NUMA node works. It is based on OvS 2.9.

Continue reading “Debugging Memory Issues with Open vSwitch DPDK”

Red Hat OpenShift

Remotely Debug an ASP.NET Core Container Pod on OpenShift with Visual Studio

Last year, I wrote a blog post how to remotely debug your ASP.NET Core container on OpenShift with Visual Studio Code. Today I introduce how to remotely debug a pod using Visual Studio from your Windows computer. Sometimes you encounter an issue that happens only in the production environment. Remotely debugging a pod enables you to investigate such an issue.

Visual Studio and Visual Studio Code now support SSH as a transport protocol for remote debugging. If a remote host accepts an SSH connection, Visual Studio can do remote debugging using Visual Studio’s default feature. However, you need to use the oc command instead of an SSH client such as putty since Red Hat OpenShift pods don’t allow direct connections via SSH. The MIEngine debugger enables you to use any command for SSH connection.

All the steps below have been confirmed using a combination of Visual Studio 2017 (versions 15.7.2 and 15.8 preview2) on Windows 10 and OpenShift 3.9.

Continue reading “Remotely Debug an ASP.NET Core Container Pod on OpenShift with Visual Studio”


Making the Operation of Code More Transparent and Obvious with SystemTap

You can study source code and manually instrument functions as described in the “Use the dynamic tracing tools, Luke” blog article, but why not make it easier to find key points in the software by adding user-space markers to the application code? User-space markers have been available in Linux for quite some time (since 2009). The inactive user-space markers do not significantly slow down the code. Having them available allows you to get a more accurate picture of what the software is doing internally when unexpected issues occur. The diagnostic instrumentation can be more portable with the user-space markers, because the instrumentation does not need to rely on instrumenting particular function names or lines numbers in source code. The naming of the instrumentation points can also make clearer what event is associated with a particular instrumentation point.

For example, Ruby MRI on Red Hat Enterprise Linux 7 has a number of different instrumentation points made available as a SystemTap tapset. If SystemTap is installed on the system, as described by What is SystemTap and how to use it?, the installed Ruby MRI instrumentation points can be listed with the stap -L” command shown below. These events show the start and end of various operations in the Ruby runtime, such as the start and end of garbage collection (GC) marking and sweeping.

Continue reading “Making the Operation of Code More Transparent and Obvious with SystemTap”


“Use the dynamic tracing tools, Luke”

A common refrain for tracking down issues on computer systems running open source software is “Use the source, Luke.” Reviewing the source code can be helpful in understanding how the code works, but the static view may not give you a complete picture of how things work (or are broken) in the code. The paths taken through code are heavily data dependent. Without knowledge about specific values at key locations in code, you can easily miss what is happening. Dynamic instrumentation tools, such as SystemTap, that trace and instrument the software can help provide a more complete understanding of what the code is actually doing

I have wanted to better understand how the Ruby interpreter works. This is an opportunity to use SystemTap to investigate Ruby MRI internals on Red Hat Enterprise Linux 7. The article What is SystemTap and how to use it? has more information about installing SystemTap. The x86_64 RHEL 7 machine has ruby-2.0.0648-33.el7_4.x86_64.rpm installed, so the matching debuginfo RPM is installed to provide SystemTap with information about function parameters and to provide me with human-readable source code. The debuginfo RPM is installed by running the following command as root:

Continue reading ““Use the dynamic tracing tools, Luke””


Using Byteman to Find Out Why the TimeZone Changed on a Java App Server

This article is about a real problem I faced where the timezone on a Java application server (in my case it was JBoss) changed unexpectedly during the run time of the server. It was hard to find any pattern or the reason for the change, as it was triggered by a HTTP request. To debug this scenario, I used the Byteman tool and injected the script into the JVM. This helped me to identify the root cause of the issue and come up with a few Do’s and Don’ts for a shared JVM (like on Java application servers).

Any application server is considered a shared JVM. There are multiple applications deployed on the JVM and they share the same resources. In such a scenario, there are some precautions which need to be taken care of. One of them is dealing with the JVM’s timezone.

Byteman is a tool that makes it easy to trace, monitor, and test the behavior of Java applications and the JDK runtime code. It injects Java code into your application APIs or into Java runtime methods without the need for you to recompile, repackage, or even redeploy your application. Injection can be performed at startup or in running code.

Continue reading “Using Byteman to Find Out Why the TimeZone Changed on a Java App Server”


Enabling Byteman Script with Red Hat JBoss Fuse and AMQ – Part 2

In my previous article, Enabling Byteman Script with Red Hat JBoss Fuse and AMQ – Part 1, we found a basic use-case for Byteman scripts with Red Hat JBoss Fuse or Red Hat JBoss AMQ. However, the log file was generated separately and only limited operations were possible. In this article I will show you how to use a Java helper class. By using Java, we get advanced operations to view or modify the content. Also, using java.util.logging allows us to log the statements to fuse.log, avoiding the creation of any other log file.

Continue reading “Enabling Byteman Script with Red Hat JBoss Fuse and AMQ – Part 2”


The GDB Python API

GDB has evolved in the last several years to provide a Python API. This series of articles will look at how a user can program GDB with the API and will also take an in-depth look at several features of that API. But, before we begin, a small history lesson is needed and a look at just why an API was needed.

Continue reading “The GDB Python API”


Programmatic Debugging: Part 1 the challenge

As every developer knows, debugging an application can be difficult and often enough you spend as much or more time debugging an application as originally writing it. Every programmer develops their collection of tools and techniques. Traditionally these have included full-fledged debuggers, instrumentation of the code, and tracing and logging. Each of these has their particular strengths and weaknesses.

Continue reading “Programmatic Debugging: Part 1 the challenge”


Debugging .NET on Red Hat Enterprise Linux from Visual Studio

Being able to edit your C# (or F# for that matter) code on your Linux VM from Visual Studio in Windows is pretty great. As a Windows developer, you’re able to work in an environment you know and trust while still being able to experiment — and hopefully produce production code — in Linux, where you may not be quite up to speed. Yet. Visual Studio, that familiar, productive and helpful IDE, is at your fingertips even though your code is far away in a Linux VM.

(Okay, not that far; the VM is running on the Red Hat Container Development Kit on your Windows box, but let’s call that last sentence “poetic license” and move on.)

Editing isn’t enough; we all know that. Since code sometimes has bugs (cough), we need to debug our code. Sure, we can write to a log and check inputs and outputs against expected results, but even that’s not enough. Any kinds of tests, even automated tests (you are using TDD, right? Right?) only confirm the existence — as rare as they are in your code — of defects. Finding the cause of a defect, ahhhh … there’s the rub. If only we could run interactive debugging from within Visual Studio against our compiled C# code running in our RHEL VM.

Well guess what; We can. It’s called “offroad debugging” and it’s not difficult to set up and — this is a technical term — it’s very cool. It’s a bit of work to get set up — not difficult, just several steps — but after doing it one time it becomes very easy for future projects. And never forget; if you get stuck, you can reach me on Twitter (@DonSchenck) or email me at

Continue reading “Debugging .NET on Red Hat Enterprise Linux from Visual Studio”