Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Using virtual functions with DPDK on Red Hat OpenShift

August 27, 2021
Wuxin Zeng
Related topics:
Containers
Related products:
Red Hat OpenShift

Share:

    For many years, organizations have optimized their networking hardware by running multiple functions and containers on Single Root I/O Virtualization (SR-IOV) network devices. The SR-IOV specification assigns a portion of a network interface card (NIC) or another device to a Kubernetes pod, so that you can share the same physical NIC among multiple pods while giving the pods direct access to the network. Organizations also use the Data Plane Development Kit (DPDK) to accelerate network traffic. This article shows you how to set up SR-IOV and DPDK on Red Hat OpenShift and run virtual functions in that environment.

    Configuring SR-IOV on OpenShift

    You can set up SR-IOV either by editing configuration files or by using the OpenShift web console, as shown in the sections that follow. In either case, you have to create a namespace, an OperatorGroup, and a Subscription.

    Editing the configuration files

    First, create a namespace for the SR-IOV Operator. In this example, we give our namespace the name openshift-sriov-network-operator and assign it a run level of 1 through the openshift.io/run-level label:

    apiVersion: v1 
    kind: Namespace 
    metadata: 
      name: openshift-sriov-network-operator 
      labels: 
        openshift.io/run-level: "1" 

    Next, create the OperatorGroup and bind it to the namespace we've just created:

    apiVersion: operators.coreos.com/v1 
    kind: OperatorGroup 
    metadata: 
      name: sriov-network-operators 
    namespace: openshift-sriov-network-operator 
    spec: 
      targetNamespaces: 
      - openshift-sriov-network-operator 
    

    Then, create the Subscription on the SR-IOV Operator:

    apiVersion: operators.coreos.com/vialpha1 
    kind: Subscription 
    metadata: 
      name: openshift-sriov-network-operator-subscription 
      namespace: openshift-sriov-network-operator 
    spec: 
      channel: "4.4" 
      name: sriov-network-operator 
      source: redhat-operators 
      sourceNamespace: openshift-marketplace 
    

    Using the web console

    Instead of editing configuration files directly, you can do the configuration through the OpenShift console.

    The demonstration in Figure 1 shows how to create a namespace object. If you use the Create Project button to create the namespace, you will not be able to name it openshift-sriov-network-operator because OpenShift does not allow you to create projects with names starting with openshift-. You can work around the limitation by creating a namespace object.

    How to use the OpenShift web console to create a namespace object.
    Figure 1: Creating a namespace object using the OpenShift web console.

    Then, as shown in Figure 2, you can go to the OperatorHub to install the SR-IOV Network Operator, which creates both the OperatorGroup and the Subscription.

    How to use the OpenShift web console to create the OperatorGroup and Subscription.
    Figure 2: Creating the OperatorGroup and Subscription using the web console.

    The Node Feature Discovery Operator

    The SR-IOV Network Node Policy requires the following node selector in the configuration file:

    feature.node.kubernetes.io/network-sriov.capable: "true"

    You can install the Node Feature Discovery (nfd) Operator to automatically label nodes so that you don’t need to add them manually. The Node Feature Discovery Operator allows you to find and label resources in all nodes and to include the node selector that will be required in configuring the network node policy. This operator finds nodes that are ready to support workloads with SR-IOV ports. Use the following YAML to install the operator:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: nfd
      namespace: openshift-operators
    spec:
      channel: "4.4"
      name: nfd
      source: redhat-operators
      sourceNamespace: openshift-marketplace

    Next, use the Node Feature Discovery Operator to create a custom resource definition (CRD) that will be managed by the operator. An example of the node feature discovery CRD follows:

    apiVersion: nfd.openshift.io/v1alpha1
    kind: NodeFeatureDiscovery
    metadata:
      name: nfd-master-server
      namespace: openshift-operators
    spec:
      namespace: openshift-nfd

    You can check that the operator is working by reviewing the node labels as follows:

    $ oc get node --show-labels 

    You can also use the console to validate the node labels, as shown in Figure 3. In the console, select Compute—>Nodes, select the node, and scroll down to see the list of node labels.

    How to use the OpenShift web console to validate node labels.
    Figure 3: Using the OpenShift web console to validate node labels.

    Configuring NICs and virtual functions

    The next configuration task is to configure the NICs that will be allocable to provide SR-IOV ports. You need to specify the physical NIC as well as the number of virtual functions that can be used per NIC.

    Not all NICs have the same number of virtual functions, so you need first to check the maximum number of virtual functions for each of your NICs. Use the Network Node State CRD in the SR-IOV Network Operator to provide the information. Enter the following command to run the operator:

    $ oc get sriovnetworknodestate -n openshift-sriov-network-operator <node name> -o yaml

    Only the SR-IOV capable NICs have virtual functions. Other NICs will show none.

    Alternatively, you can review the SR-IOV capable NICs in the web console, as shown in Figure 4. Go to Installed Operators, select the Sriov network state, click on a Sriov network node, and click on YAML to see the details.

    How to review the SR-IOV capable NICs in the OpenShift web console.
    Figure 4: Reviewing SR-IOV capable NICs in the OpenShift web console.

    Now select the SR-IOV capable NICs that you want to use in the environment. For this selection, you can use the SRIOV network node policy custom resource definition created by the SR-IOV Network Operator. You can include the name of the NIC physical function and the range of virtual functions to be used in the nicSelector specification, in the following format:

    <pfname>#<first_vf>-<last_vf>

    You can change the driver type for the virtual functions in the deviceType specification, which allows a choice of netdevice or vfio-pci for each virtual function. The netdevice option performs the device binding in kernel space, whereas vfio-pci does the binding in user space. The default value is netdevice. However, because we’re using DPDK in this example, we will be working in user space, so we'll select vfio-pci for the device type to perform the binding in user space.

    Figure 5 shows how to select the device type using the console. Select the SR-IOV Network Operator, and create an SR-IOV network node policy instance.

    How to use the OpenShift web console to select the device type.
    Figure 5: Selecting the device type in the web console.

    The following YAML file shows that the NIC selection is done per the SR-IOV network node policy:

    apiVersion: sriovnetwork.openshift.io/v1
    kind: SriovNetworkNodePolicy
    metadata:
      name: policy-pcie
      namespace: openshift-sriov-network-operator 
    spec:
      resourceName: pcieSRIOV
      nodeSelector:
        feature.node.kubernetes.io/network-sriov.capable: "true" 
      mtu: 1500 
      numVfs: 64 
      nicSelector: 
        pfNames: ["ens1f0#0-49", "ens1f1#0-49"] 
      deviceType: netdevice
      isRdma: false

    Configuring the network

    Next, you need to configure the network that will be attached to your SR-IOV ports in order to have SR-IOV working in your environment. When you configure the network, you can customize some aspects. If you don’t have a DHCP server in your network, you can define static IP addresses on your pods and enable static IP capability. Also, make sure the namespace is the one where you’ll be using the SR-IOV network:

    apiVersion: sriovnetwork.openshift.io/v1
    kind: SriovNetwork
    metadata:
      name: sriovnet1
      namespace: openshift-sriov-network-operator 
    spec:
      ipam: |
        {
          "type": "static", 
          "addresses": [ 
            {
              "address": "192.168.99.0/24", 
              "gateway": "192.168.99.1" 
            }
          ],
          "dns": { 
            "nameservers": ["8.8.8.8"], 
            "domain": "test.lablocal", 
            "search": ["test.lablocal"] 
          }
        }
      vlan: 0
      spoofChk: 'on'
      trust: 'off'
      resourceName: onboardSRIOV
      networkNamespace: test-epa
      capabilities: '{"ips": true}'

    With these configuration changes, you should have SR-IOV working in your environment. To test your configuration, create two pods and add a secondary network using the SR-IOV network. You will also need to configure static IP addresses in both pods. Add nodeName definitions to force the pods to run in different worker nodes. Once the pods are running, just ping one pod from the other.

    Configuring DPDK on OpenShift

    Configuring DPDK follows the same steps for configuring SR-IOV, with a few modifications. First, DPDK requires configuring huge pages along with the SR-IOV configuration. And, as mentioned earlier, when using DPDK, you need to select a device type of vfio-pci to bind the device in user space.

    An example for an Intel NIC follows:

    Note: If you’re using a Mellanox NIC, you must use the netDevice driver type and set isRdma to true:

    apiVersion: sriovnetwork.openshift.io/v1
    kind: SriovNetworkNodePolicy
    metadata:
      name: policy-onboard-dpdk
      namespace: openshift-sriov-network-operator 
    spec:
      resourceName: onboardDPDK
      nodeSelector:
        feature.node.kubernetes.io/network-sriov.capable: "true" 
      mtu: 1500 
      numVfs: 64
      nicSelector: 
        pfNames: ["eno5#50-59", "eno6#50-59"] 
      isRdma: false
      deviceType: vfio-pci

    The SR-IOV network object configuration for DPDK is the same as the one used for a "plain" SR-IOV configuration:

    apiVersion: sriovnetwork.openshift.io/v1
    kind: SriovNetwork
    metadata:
      name: dpdknet1
      namespace: openshift-sriov-network-operator 
    spec:
      ipam: |
        {
          "type": "static", 
          "addresses": [ 
            {
              "address": "192.168.155.0/24", 
              "gateway": "192.168.155.1" 
            }
          ],
          "dns": { 
            "nameservers": ["8.8.8.8"], 
            "domain": "testdpdk.lablocal", 
            "search": ["testdpdk.lablocal"] 
          }
        }
      vlan: 0
      spoofChk: 'on'
      trust: 'off'
      resourceName: onboardDPDK
      networkNamespace: test-epa
      capabilities: '{"ips": true}'

    Lastly, to test DPDK, use an image that uses DPDK to create a pod that includes both the huge pages configuration and the SR-IOV network definition.

    Conclusion

    In this article, I have tried to show that the popular and powerful SR-IOV and DPDK capabilities of network devices are easy to set up on OpenShift. The web console simplifies many tasks, and using operators automates a lot of the configuration. Try these features on your own projects to save resources and increase the capabilities of your network hardware.

    Last updated: November 6, 2023

    Related Posts

    • Can you run Intel's Data-plane Development Kit (DPDK) in a Docker container? Yep.

    • How to switch Red Hat OpenShift Virtualization from hardware virtualization to software emulation

    • OVS-DPDK Parameters: Dealing with multi-NUMA

    • Debugging Memory Issues with Open vSwitch DPDK

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue