It isn’t always easy to understand how Open vSwitch (OVS) cycles are spent, especially because various parameters and configuration options can affect how OVS behaves. Members of the Open vSwitch community are actively working to understand what causes packets drops in Open vSwitch. Efforts so far have included adding a custom statistic for vHost TX retries, tracking vHost TX contention, and adding a coverage counter to count vHost IRQs. We are particularly interested in the user space datapath that uses the Data Plane Development Kit (DPDK) for fast I/O.
Adding these statistics is an ongoing effort, and we won’t cover all of the corners. In some cases, the statistics leave doubts about what is causing a behavior.
In this article, I will introduce a new counter we’ve added to learn more about contention in the vHost transmission path. I’ll also show you how to use the new counter with
perf, and I’ll discuss what’s next for our ongoing efforts.
Continue reading “Debugging vHost user TX contention in Open vSwitch”
Open vSwitch (OVS) can use the kernel datapath or the userspace datapath. There are interesting developments in the kernel datapath using hardware offloading through the TC Flower packet classifier, but in this article, the focus will be on the userspace datapath accelerated with the Data Plane Development Kit (DPDK) and its new feature—partial flow hardware offloading—to accelerate the virtual switch even more.
This article explains how the virtual switch worked before versus now and why the new feature can potentially save resources while improving the packet processing rate.
Continue reading “Speeding up Open vSwitch with partial hardware offloading”
The most common problem when people are trying to deploy an Open vSwitch with Data Plane Development Kit (OvS-DPDK) solution is that the performance is not as expected. For example, they are losing packets. This is where our journey for this series of blogs will start.
This first blog is about Poll Mode Driver (PMD) thread core affinity. It covers how to configure thread affinity and how to verify that it’s set up correctly. This includes making sure no other threads are using the CPU cores.
Continue reading “Troubleshooting Open vSwitch DPDK PMD Thread Core Affinity”
This article is about debugging out-of-memory issues with Open vSwitch with the Data Plane Development Kit (OvS-DPDK). It explains the situations in which you can run out of memory when using OvS-DPDK and it shows the log entries that are produced in those circumstances. It also shows some other log entries and commands for further debugging.
When you finish reading this article, you will be able to identify that you have an out-of-memory issue and you’ll know how to fix it. Spoiler: Usually having some more memory on the relevant NUMA node works. It is based on OvS 2.9.
Continue reading “Debugging Memory Issues with Open vSwitch DPDK”
In a few weeks, the Fast Datapath Production channel will update the Open vSwitch version from the 2.7 series to the 2.9 series. This is an important change in more ways than one. A wealth of new features and fixes all related to packet movement will come into play. One that will surely be blamed for all your troubles will be the integration of the `–ovs-user` flag to allow for an unprivileged user to interact with Open vSwitch.
Running as root can solve a lot of pesky problems. Want to write to an arbitrary file? No problem. Want to load kernel modules? Go for it! Want to sniff packets on the wire? Have a packet dump. All of these are great when the person commanding the computer is the rightful owner. But the moment the person in front of the keyboard isn’t the rightful owner, problems occur.
Continue reading “Non-root Open vSwitch in RHEL”
In order to maximize performance of the Open vSwitch DPDK datapath, it pre-allocates hugepage memory. As a user you are responsible for telling Open vSwitch how much hugepage memory to pre-allocate. The question of exactly what value to use often arises. The answer is, it depends.
There is no simple answer as it depends on things like the MTU size of the ports, the MTU differences between ports, and whether those ports are on the same NUMA node. Just to complicate things a bit more, there are multiple overheads, and alignment and rounding need to be accounted for at various places in OVS-DPDK. Everything clear? OK, you can stop reading then!
However, if not, read on.
Continue reading “Open vSwitch-DPDK: How Much Hugepage Memory?”
In Network Function Virtualization, there is a need to scale functions (VNFs) and infrastructure (NFVi) across multiple NUMA nodes in order to maximize resource usage.
In this blog, we’ll show how to configure Open vSwitch using DPDK datapath (OVS-DPDK) parameters for multiple NUMA systems, based on OVS 2.6/2.7 using DPDK 16.11 LTS.
Continue reading “OVS-DPDK Parameters: Dealing with multi-NUMA”