Skip to main content
Redhat Developers  Logo
  • Products

    Platforms

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat AI
      Red Hat AI
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • View All Red Hat Products

    Featured

    • Red Hat build of OpenJDK
    • Red Hat Developer Hub
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat OpenShift Local
    • Red Hat Developer Sandbox

      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Secure Development & Architectures

      • Security
      • Secure coding
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • Product Documentation
    • API Catalog
    • Legacy Documentation
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Example-CNF: Automating the deployment of DPDK-based network functions on OpenShift with fault tolerance

December 17, 2024
Ramon Perez Manuel Rodriguez Hernandez
Related topics:
Automation and managementKubernetesOperators
Related products:
Red Hat Ansible Automation PlatformRed Hat OpenShift Container Platform

Share:

    Network optimization with Data Plane Development Kit (DPDK) and Single Root Input/Output Virtualization (SR-IOV) is widely supported and adopted in Kubernetes and Red Hat OpenShift. In this article, we will present a means of automatically deploying an open source Cloud-Native Network Function (CNF), called Example-CNF, in order to test particular Telco-related configurations and scenarios, leveraging DPDK's capabilities within containerized environments.

    Example-CNF is a workload to exercise both DPDK and SR-IOV technologies for Telco setups. The project is composed of several Kubernetes operators built with open source toolkits, such as Operator SDK, using Red Hat Universal Base Images (UBI) images pushed to Quay.io, deploying two main components:

    • A packet generator, based on TRex, which emulates different 5G User Equipment (UE) network traffic profiles.
    • A traffic forwarder, based on TestPMD (an example application shipped in DPDK) that manages the traffic sent by the packet generator, forwarding it back to it and computing packet loss.

    The network scenario is shown in Figure 1, where we can see the following:

    • Each component is deployed in a different worker node, thanks to pod anti-affinity rules.
    • Each component has two ports, each of them connected to a virtual network interface controller (NIC) managed by SR-IOV. Each NIC belongs to a given SR-IOV network, related to a given network configuration: physical interface, VLAN, and etc.
    Example-CNF network diagram
    Figure 1: Example-CNF network diagram.

    Validated use cases range from a simple TRex-DPDK traffic flow to more complex and end-to-end cases where cluster nodes are cordoned and drained (emulating a cluster upgrade), thus ensuring fault tolerance in the workflow while nodes get reconciliation. All these scenarios have been automated with Red Hat Ansible Automation Platform’s playbooks and roles that validate some prerequisites and prepare the environment to orchestrate the installation and testing, all being managed by Red Hat Distributed-CI.

    What do you need to launch Example-CNF?

    In a nutshell, you need to meet the following requirements:

    • Have an OpenShift cluster composed by +3 worker nodes.
    • Install SR-IOV Network Operator.
    • Deploy SriovNetworkNodePolicy and SriovNetwork CustomResourceDefinitions (CRDs) that describe your network setup.
      • Each node needs a physical NIC with at least two SR-IOV Virtual Functions.
      • Each SRIOV network requires a VLAN.
    • Make a proper configuration in network devices (switches, nodes, etc.) to match the SR-IOV resources definition (physical interfaces, VLAN, etc.)
    • Deploy a PerformanceProfile, setting up CPU isolation, hugepages, etc.

    How are Operators automated?

    Example-CNF is composed by three main Operator Lifecycle Manager (OLM)-based operators:

    • trex-operator: Packet generator, based on TRex, which emulates different 5G-UE network traffic profiles targeting specific CNFs.
    • testpmd-operator: TestPMD-based CNF that performs traffic forwarding.
    • cnf-app-mac-operator: Extra custom resource that retrieves MAC addresses from TestPMD pods to make them available to other resources.

    All these operators have been built with the Operator SDK framework, which relies on the operator reconciliation loop. This feature is explained in Figure 2.

    How operator reconciliation loop works in Operator SDK
    Figure 2: How operator reconciliation loop works in Operator SDK.

    The key topic is how Kubernetes/OpenShift resources are created (for example, a deployment with two pods), based on the generic description provided by the CustomResourceDefinition object. The link between them is the CustomResource, which is the object that is observed by the reconciliation loop. This way, when a CustomResource is instantiated in the cluster, this eventually implies the creation of the desired cluster resources. In this video, you can see how TestPMD Pods are deployed in Example-CNF, using the TestPMD CustomResource.

    These three operators are working together to reach the final objective of deploying and testing all pieces of software. This is summarized in Figure 3.

    Example-CNF lifecycle workflow
    Figure 3: Example-CNF lifecycle workflow.

    Key details of this workflow are the following:

    • Image generation is automated with Github workflows in the example-cnf repository.
    • The example_cnf_deploy public Ansible role is used to deploy Example-CNF.
    • The workflow involves several Kubernetes resources.
    • The process is orchestrated with Red Hat Distributed-CI.

    Testing and validation

    The example_cnf_deploy public Ansible role is also used for testing Example-CNF. In particular, there are different scenarios that are covered under this automation:

    • Simple TRex-TestPMD traffic flow (shown in Figure 1).
    • Validation during cluster upgrades, where cluster nodes are cordoned and drained, thus ensuring fault tolerance in the workflow while nodes get reconciliation.

    For this second scenario, the automation follows the following steps, that are also presented in this video:

    • Deploy TestPMD and TRex.
    • Launch a TRex test with enough duration to perform node cording-draining.
    • Cordon-drain the node where TestPMD is running.
    • Wait until TestPMD is allocated in a different worker node.
    • Measure the following statistics:
      • Downtime: Time that TestPMD spends for being recreated in a new worker node.
      • Packet loss: How many packets were lost, and what’s the percentage of packet loss in the test case. For the calculation, TestPMD and TRex counters are read with some scripts.

    Conclusions and future work

    In this article, we have discussed the following:

    • The benefits of using an approach based on Kubernetes operators and Ansible roles and playbooks. This approach facilitates the testing of new changes with CI, and it accelerates the integration when new versions of tools are available for deployment and testing.
    • Specific examples about how to build automation around TestPMD and TRex.

    However, we were not free of challenges found. In particular, we found complications when trying to automate the TestPMD configuration, since the two different modes that it has to be launched (auto-start and interactive) cannot be used simultaneously, which made debugging and troubleshooting harder to follow, because sometimes we were interested in checking on live the status of TestPMD pod statistics, but if not using interactive mode, it was hard to automate.

    To deal with these issues, the idea of future work is to make the DPDK piece more (and better) configurable. The idea is to replace TestPMD with Grout, an open source DPDK-based network processing application. Example-CNF would benefit from the message-based API offered by Grout to be able to simultaneously use multiple ways of configuring the tool: interactive shell, script, batches, and etc.

    The final objective is to define a CNF emulating 5G User Plane Functions that supports cluster upgrades seamlessly, being able to modify the DPDK configuration whenever needed, and achieving more complex use cases with the usage of Grout, such as load-balancing through the fabric switch.

    Last updated: January 8, 2025

    Related Posts

    • Orchestrate offloaded network functions on DPUs with Red Hat OpenShift

    • Debugging Memory Issues with Open vSwitch DPDK

    • Improve multicore scaling in Open vSwitch DPDK

    • OVS-DPDK Parameters: Dealing with multi-NUMA

    Recent Posts

    • How to deploy language models with Red Hat OpenShift AI

    • AI search with style: Fashion on OpenShift AI with EDB

    • What qualifies for Red Hat Developer Subscription for Teams?

    • How to run OpenAI's gpt-oss models locally with RamaLama

    • Using DNS over TLS in OpenShift to secure communications

    What’s up next?

    Explore essential Ansible concepts, terminology, and tools, then set up your environment to start writing  your first Ansible playbook in this hands-on learning path.

    Start the activity
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue