As part of our participation in hundreds of open source communities, Red Hat engineers are often involved in research and development efforts that may or may not become a part of Red Hat's supported offerings.
Intel's Data-plane Development Kit (DPDK) is a set of libraries and drivers for Linux and BSD built for fast packet processing, for the burgeoning "Network Function Virtualization", or NFV discipline. Typical verticals interested in turning Linux boxes into packet-processing machines are telecom, financial services, military, energy research, datacenter operators, internet service providers and many more.
These applications often have very high performance requirements for both throughput, latency, and cycles-per-packet (a measure of transmission "efficiency"), that users are willing to trade-off the maturity and feature-set of an operating system kernel to meet their needs in particular use-cases
DPDK is the newest in a decades-long line of kernel-bypass technologies that bring the network stack into userspace, allowing network adapters to DMA into application memory.
Other examples of these technologies are:
Several network adapter vendors have also built kernel-bypass technologies optimized for their hardware:
All of these technologies work just fine with RHEL, and are in production at a large number of performance-sensitive sites around the world.
Last month, Red Hat released a whitepaper called Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnload. This paper was the first to document and prove that extreme low latency solutions can be built using Linux containers and kernel-bypass technology. It was accompanied by a webinar discussing the techniques and results.
The rapidly evolving NFV space has some of our industry's most demanding performance requirements all on commodity, x86 gear. For example:
- Forwarding tens-of-millions of packets per second, per CPU core
- Jitter measured in single-digit microseconds
- Extremely tight standard deviation
Making it all happen on commodity hardware is where Linux shines, as de facto operating system of choice for high performance computing. Can we achieve the same performance levels as purpose-built hardware, when using off-the-shelf, commodity servers and software?
It turns out that we can achieve bare-metal performance levels when running DPDK sample applications such as l2fwd, using techniques and automation documented in the docker-dpdk github repo. We used dpdk-pktgen to generate the load on Haswell-based systems with i40e NICs.
The approach is as follows:
- Use the rhel7/rhel-tools base container image, because we are going to compile DPDK from source in the process of building the container. This is only necessary for demos when DPDK must be rebuilt for different kernels. In a production scenario, you may be able to drop a compiled version of DPDK into a container and go with the base RHEL 7 container (without tools, compilers and toolchain).
- Run the container with elevated privileges (see the Dockerfile for a precise example) so that it has direct access to certain system resources required by the DPDK stack, such as kernel module manipulation, access to hotplug PCI, hugepages, and write access to NUMA topology within sysfs (so the DPDK scripts can allocate both 2M and 1G hugepages properly).
- We also have the same chicken-and-egg problem we have with the Solarflare OpenOnload container: when to insert the kernel module. If we insert it before container creation, that means DPDK etc must be installed on the host. If we insert it after container creation, then we have to pass in /dev as a volume. Neither approach is optimal, and we're working on elegant solutions.
- Also included in the github repo is some example tuning suggestions to get improve determinism and reduce jitter. These tunings are further documented in the Low Latency Tuning Guide for RHEL7.
As is the case with most of things related to container technologies, this is all very much a work in progress. Mixing next generation infrastructure such as containers, with next generation network function virtualization is just one of the areas that Red Hat is pursuing through efforts in OpenStack, Open Platform for NFV (OPNFV), Project Atomic and others.
The impressive performance achievable with Linux, containers and technologies such as DPDK puts the vision of a superior, fully open source packet-processing stack well within reach.
Last updated: January 13, 2023