Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Achieving high-performance, low-latency networking with XDP: Part I

 

December 6, 2018
Paolo Abeni
Related topics:
C, C#, C++Linux
Related products:
Red Hat Enterprise Linux

Share:

    XDP: From zero to 14 Mpps

    In past years, the kernel community has been using different approaches in the quest for ever-increasing networking performance. While improvements have been measurable in several areas, a new wave of architecture-related security issues and related counter-measures has undone most of the gains, and purely in-kernel solutions for some packet-processing intensive workloads still lag behind the bypass solution, namely Data Plane Development Kit (DPDK), by almost an order of magnitude.

    But the kernel community never sleeps (almost literally) and the holy grail of kernel-based networking performance has been found under the name of XDP: the eXpress Data Path. XDP is available in Red Hat Enterprise Linux 8, which you can download and run now.

    This technology allows you to implement new networking features (and/or re-implement existing ones) via custom extended BPF (eBPF) programs attached to the kernel packet processing deep down the stack, killing the overhead of socket buffers (SKBs) management, reducing the per-packet memory management overhead, and allowing more-effective bulking. XDP eBPF programs have access to helpers for packet manipulation and packet forwarding, offering almost unlimited opportunity to change and extend the kernel behavior without the need to add new in-kernel code—and, while at it, reaching a higher possible processing speed. A modern driver with XDP support can easily handle more than 14 Mpps.

    Excited by this opportunity, but scared by the unknown? This article will guide you towards your first XDP program, building a working example from zero and allowing you to build a light-speed network application from there.

    XDP overview

    XDP allows you to attach an eBPF program to a lower-level hook inside the kernel. Such a hook is implemented by the network device driver, inside the ingress traffic processing function (usually, the NAPI poll() method), before an SKB is allocated for the current packet.

    The program entry function has a single argument: a context describing the current packet. The program can manipulate such packet in arbitrary ways, but it has to respect the constraints imposed by the eBPF verifier (a little more on this later). Finally, such program must return a controlling action to the owning device driver, specifying how the device driver should cope next with the processed packet, for example, passing it to the upper-layer processing or dropping it.

    The eBPF verifier imposes some restrictions on the code you can write, performing rigorous checks to ensure that the program:

    • contains no loop
    • accesses only valid memory (for example, does not exceed the packet boundaries)
    • uses a limited number of eBPF instructions, since the limit in the Linux 4.14 kernel is 128K

    The verifier itself runs inside the kernel and is executed when the eBPF program is loaded via the eBPF syscall. Once the program has been successfully loaded, it can be attached to the XDP hook for any number of devices. The kernel source bundles a user-space library (libbpf) with helper functions to simplify such tasks.

    Adventurous (or advanced) users can write an eBPF program directly using the eBPF assembler, but it's much easier to use a higher-level programming language and let the compiler translate the code to eBPF. The eBPF community selected the LLVM compiler and the C language for such a task. Since the verifier takes action on compiled code, any optimization performed by the compiler can affect the results. For example, loops bound to a constant number of iterations are unrolled by the compiler. This allows you to circumvent the "no loops" constraint, but it also increases the number of eBPF instructions generated; nested loops can easily hit the 128K limit.

    XDP flavors

    Not all network device drivers implement the XDP hook. In such a case, you may fall back to the generic XDP hook, implemented by the core kernel and available regardless of the specific network device driver feature. Since such a hook takes place later in the networking stack, after SKB allocation, the performance observed there is much lower than the driver-based XDP hook, but it still allows experimenting with XDP. The network device drivers supporting the XDP hook in Linux 4.18 and later are:

    • bnxt
    • thunder
    • i40e
    • ixgbe
    • mlx4
    • mlx5
    • nfp
    • qede
    • tun
    • virtio_net

    The list can vary with the kernel version, so it's worth checking for explicit support for your own driver in the running kernel; see below for how to do that.

    Finally, the XDP/eBPF program behavior can be controlled and inspected by the user-space via a number of maps defined and used by the program. Such maps are accessible and modifiable also by the user-space, still using the libbpf helpers.

    "Hello world"

    XDP is not a programming language, butit it uses a programming language (eBPF), and every programming-related tutorial has to start with a "Hello world" program. While we can somewhat echo debug messages to XDP/eBPF program—and we will, later—we'll start instead from an even simpler XDP example: an eBPF program that does almost nothing, just passing each processed packet up to the kernel stack.

    // SPDX-License-Identifier: GPL-2.0
    
    #define KBUILD_MODNAME "xdp_dummy"
    #include <uapi/linux/bpf.h>
    #include "bpf_helpers.h"
    
    SEC("proc")
    int xdp_dummy(struct xdp_md *ctx)
    {
        return XDP_PASS;
    }
    
    char _license[] SEC("license") = "GPL";
    

    Let's look at the details for the code above. It uses C-language syntax and includes two external headers. The first one, provided by the kernel-header package of most distributions, contains the definition of the XDP program return code, XDP_PASS in this example, meaning that the kernel will pass along the packet to network processing. Other available values are XDP_DROP (does the same), XDP_TX (sends back the packet out of the interface that received it), XDP_REDIRECT (sends the packet out of another interface).

    The second header, which still is part of the Linux kernel but is not usually packaged by most distributions, contains a list of the available eBPF helpers and
    the definition of the SEC() macro. The latter is used to place a fragment of the compiled object in different ELF sections. Such sections will be interpreted by
    the eBPF loader to detect, for example, the maps defined by the program (and allow user-space access to them).

    Finally, the last line formally specifies the license associated with this program. Some eBPF helpers are accessible only by GPLed programs, and the verifier will use this info to enforce such a restriction.

    Some rough edges

    Let's build and run it! LLVM/Clang version 3.7 or later (version 6.0 was used for this writing, the Fedora 28 default) and a reasonable, not-obsolete
    make version (here, 4.2.1) are needed. This is the makefile used:

    KDIR ?= /lib/modules/$(shell uname -r)/source
    CLANG ?= clang
    LLC ?= llc
    ARCH := $(subst x86_64,x86,$(shell arch))
    
    BIN := xdp_dummy.o
    CLANG_FLAGS = -I. -I$(KDIR)/arch/$(ARCH)/include \
    -I$(KDIR)/arch/$(ARCH)/include/generated \
    -I$(KDIR)/include \
    -I$(KDIR)/arch/$(ARCH)/include/uapi \
    -I$(KDIR)/arch/$(ARCH)/include/generated/uapi \
    -I$(KDIR)/include/uapi \
    -I$(KDIR)/include/generated/uapi \
    -include $(KDIR)/include/linux/kconfig.h \
    -I$(KDIR)/tools/testing/selftests/bpf/ \
    -D__KERNEL__ -D__BPF_TRACING__ -Wno-unused-value -Wno-pointer-sign \
    -D__TARGET_ARCH_$(ARCH) -Wno-compare-distinct-pointer-types \
    -Wno-gnu-variable-sized-type-not-at-end \
    -Wno-address-of-packed-member -Wno-tautological-compare \
    -Wno-unknown-warning-option \
    -O2 -emit-llvm
    
    all: $(BIN)
    
    xdp_dummy.o: xdp_dummy.c
    $(CLANG) $(CLANG_FLAGS) -c $< -o - | \
    $(LLC) -march=bpf -mcpu=$(CPU) -filetype=obj -o $@
    

    It is fairly non-trivial: because the program uses kernel headers, we need to cope with architecture specific headers dependencies. Invoking make on your distribution of choice with the above makefile can be disappointing, as you will probably get something like this:

    clang -nostdinc -I. \
    [... long argument list elided for clarity ...]
    -O2 -emit-llvm -c xdp_dummy.c -o - | \
    llc -march=bpf -mcpu= -filetype=obj -o xdp_dummy.o
    xdp_dummy.c:5:10: fatal error: 'bpf_helpers.h' file not found
    #include "bpf_helpers.h"
    ^~~~~~~~~~~~~~~
    1 error generated.
    

    Almost no distribution packages include the complete kernel sources, including the required bpf_helper.h. The current solution is, unfortunately, to download the full Linux sources, unpack them somewhere on your local disc, and invoke make as follow:

    KDIR= make
    

    Now we have our eBPF/XDP program, but we must load it inside the kernel to get any effect. For a less-trivial example, the load part is usually done by a controlling
    user-space program, which will also monitor/interact with the XDP program via some maps. For instant gratification, we can use iproute, instead:

    ip link set dev <net device name> xdp object xdp_dummy.o
    

    The above code attaches our XDP program to the specified network device, trying to use the device hook and falling back to the generic one otherwise. Those who are brave and have a supported device driver can replace xdp with xdpdrv to force the usage of the driver-specific hook.  Attaching will fail if xdpdrv is not available.

    What's next

    With your newly written XDP program, you can experience unprecedented speed in packet filtering—unless you already did that by unplugging the Ethernet cable— as a modern driver with XDP support can easily handle more than 14 Mpps! But you could be interested in doing something useful, like dropping specified packets or collecting statistics about the XDP program activity. In the next article in this series, we will deal with that, moving on to a less-trivial example by introducing packet parsing, debugging, and map usage!

    You might also be interested in this article: Using eXpress Data Path (XDP) maps in RHEL 8: Part 2.

    Last updated: March 24, 2023

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    What’s up next?

     

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue