Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Orchestrate offloaded network functions on DPUs with Red Hat OpenShift

April 26, 2022
Erwan Gallen
Related topics:
LinuxKubernetes
Related products:
Red Hat OpenShift

Share:

    The traditional CPU-centric system architecture is being replaced by designs where systems are aggregated from independently intelligent devices. These systems have their own compute capabilities and can natively run network functions with an accelerated data plane. The new model allows us to offload to accelerators not only the individual subroutines but whole software subsystems, such as networking or storage, with cloud-like security isolation and architectural compartmentalization.

    One of the most prominent examples of this new architecture is the data processing unit (DPU). DPUs offer a complete compute system with an independent software stack, network identity, and provisioning capabilities. The DPU can host its own applications using either embedded or orchestrated deployment models.

    The unique capabilities of the DPU allow for key infrastructure functions and their associated software stacks to be completely removed from the host node’s CPU cores and to be relocated onto the DPU. For instance, DPU could host the management plane of the network functions and part of the control plane, while the data plane could be accelerated by dedicated Arm cores, ASICs, GPUs, or FPGA IPs. Because DPUs can run independent software stacks locally, multiple network functions could run simultaneously on the same devices with service chaining and shared accelerators to provide generic in-line processing.

    OVN/OVS offloading on NVIDIA BlueField-2 DPUs

    Red Hat has collaborated with NVIDIA to extend the operational simplicity and hybrid cloud architecture of Red Hat OpenShift to NVIDIA BlueField-2 DPUs. Red Hat OpenShift 4.10 provides BlueField-2 OVN/OVS offload as a developer preview.

    Installing OpenShift Container Platform on a DPU makes it possible to offload packet processing from the host CPUs to the DPU. Offloading resource-intensive tasks like packet processing from the server’s CPU to the DPU can free up cycles on the OpenShift worker nodes to run more user applications. OpenShift brings portability, scalability, and orchestration to DPU workloads, giving you the ability to use standard Kubernetes APIs along with consistent system management interfaces for both the host systems and DPUs.  

    In short, utilizing OpenShift with DPUs lets you get the benefits of DPUs without sacrificing the hybrid cloud experience or adding unnecessary complexity to managing IT infrastructure.

    To manage DPUs, Red Hat OpenShift replaces the native BlueField operating system (OS) on each DPU and is deployed using a two-cluster design that consists of:

    • Tenant cluster running on the host servers (x86)
    • Infrastructure cluster running on DPUs (Arm)

    The architecture is illustrated in Figure 1.

    Diagram of the architecture.
    Figure 1: The two-cluster design.

    In this architecture, DPUs are provisioned as worker nodes of the Arm-based OpenShift infrastructure cluster. This is the blue cluster in Figure 1. The tenant OpenShift cluster, composed of the x86 servers, is where user applications typically run. This is the green cluster. In this deployment, each physical server runs both a tenant node on the x86 cores and an infrastructure node on the DPU Arm cores.

    This architecture allows you to minimize the attack surface by decoupling the workload from the management cluster.

    This architecture also streamlines operations by decoupling the application workload from the underlying infrastructure. That allows IT Ops to deploy and maintain the platform software and accelerated infrastructure while DevOps deploys and maintains application workloads independently from the infrastructure layer.

    Red Hat OpenShift 4.10 provides capabilities for offloading Open Virtual Network (OVN) and Open Virtual Switch (OVS) services that typically run on servers, from the host CPU to the DPU. We are offering this functionality as a developer preview and enabling the following components to support OVN/OVS hardware offload to NVIDIA BlueField-2 DPUs:

    • DPU Network Operator: This component is used with the infrastructure cluster to facilitate OVN deployment.
    • DPU mode for OVN Kubernetes: This component is assigned by the cluster network operator for the tenant cluster.
    • SR-IOV network operator: This component discovers compatible network devices, such as the ConnectX-6 Dx NIC embedded inside the BlueField-2 DPU, and provisions them for SR-IOV access by pods on that server
    • ConnectX NIC Fast Data Path on Arm
    • Kernel flow offloading (TC Flower)
    • Experimental use of OpenShift Assisted Installer and BlueField-2 BMC

    The combination of these components allows us to move ovn-kube-node services from the x86 host to the BlueField-2 DPU.

    The network flows are offloaded in this manner (see Figure 2):

    • The ovn-k8s components are moved from the x86 host to the DPU (ovn-kube, vswitchd, ovsdb).
    • The Open vSwitch data path is offloaded from the BlueField Arm CPU to the ConnectX-6 Dx ASIC.
    Diagram of the network flows.
    Figure 2: Illustrating the network flows.

    The following Open vSwitch datapath flows managed by ovn-k8s are offloaded to a BlueField-2 DPU that is running OpenShift 4.10:

    • Pod to pod (east-west)
    • Pod to clusterIP service backed by a regular pod in diff node (east-west)
    • Pod to external (north-south)

    Let’s take a more detailed look at the testbed setup.

    Install and configure an accelerated infrastructure with OpenShift and NVIDIA BlueField-2 DPUs

    In our sample testbed shown in Figure 3, we are using two x86 hosts with BlueField-2 DPU PCIe cards installed. Each DPU has eight Arm cores, two 25GB network ports, and a 1GbE management port. We’ve wired 25GB ports to a switch and also connected the BMC port of the DPU to a separate network to manage the device with IPMI.

    A photo of the sample testbed.
    Figure 3: The sample testbed.

    Next, we’ve installed Red Hat OpenShift Container Platform 4.10 on a DPU to offload packet processing from the host x86 to the DPU. Offloading resource-intensive computational tasks such as packet processing from the server’s CPU to the DPU frees up cycles on the OpenShift Container Platform worker nodes to run more applications or to run the same number of applications more quickly.

    OpenShift and DPU deployment architecture

    In our setup, OpenShift replaces the native BlueField OS. We used the two-cluster architecture where DPU cards are provisioned as worker nodes in the Arm-based infrastructure cluster. The tenant cluster composed of x86 servers was used to run user applications.

    We followed these steps to deploy tenant and infrastructure clusters:

    1. Install OpenShift assisted installer on the installer node using Podman.
    2. Install the infrastructure cluster.
    3. Install the tenant cluster.
    4. Install the DPU Network Operator on the infrastructure cluster.
    5. Configuring SR-IOV Operator for DPUs.

    For details about how to install the OpenShift 4.10 on BlueField with OVN/OVS hardware offload, refer to the documentation.

    When the cluster is deployed on the BlueField, the OVN configuration is automated with the "DPU Network Operator"; this operator can be installed in the OpenShift console or with the oc command.

    As shown in Figure 4, the DPU operator is available for the Arm-based OpenShift clusters in the catalog and not visible in x86 OpenShift clusters.

    The DPU Network Operator shown in the Operator Hub catalog.
    Figure 4: The DPU Network Operator shown in the OperatorHub catalog.

    Validate installation using the OpenShift console

    When you have completed the last step, you have two OpenShift clusters running. We will make some console checks to show the configuration done and benchmark the OVS/OVN offloading.

    Tenant cluster: We can list the nodes of the Tenant cluster nodes:

    [egallen@bastion-tenant ~]$ oc get nodes
    NAME                     STATUS   ROLES             AGE   VERSION
    tenant-master-0          Ready    master            47d   v1.23.3+759c22b
    tenant-master-1          Ready    master            47d   v1.23.3+759c22b
    tenant-master-2          Ready    master            47d   v1.23.3+759c22b
    tenant-worker-0          Ready    worker            47d   v1.23.3+759c22b
    tenant-worker-1          Ready    worker            47d   v1.23.3+759c22b
    x86-worker-advnetlab13   Ready    dpu-host,worker   35d   v1.23.3+759c22b
    x86-worker-advnetlab14   Ready    dpu-host,worker   41d   v1.23.3+759c22b

    Infrastructure cluster: We can list the nodes of the Infrastructure cluster, including the BlueField-2 nodes with the machine-config pool name dpu-worker:

    [egallen@bastion-infrastructure ~]$ oc get nodes
    NAME       STATUS   ROLES               AGE   VERSION
    bf-13      Ready    dpu-worker,worker   71d   v1.22.1+6859754
    bf-14      Ready    dpu-worker,worker   74d   v1.22.1+6859754
    master-0   Ready    master              75d   v1.22.1+6859754
    master-1   Ready    master              75d   v1.22.1+6859754
    master-2   Ready    master              75d   v1.22.1+6859754
    worker-0   Ready    worker              70d   v1.22.1+6859754
    worker-1   Ready    worker              70d   v1.22.1+6859754

    Tenant cluster: We can list the pods running in the openshift-ovn-kubernetes namespace in the tenant cluster:

    [egallen@bastion-tenant ~]$ oc get pods -n openshift-ovn-kubernetes 
    NAME                          READY   STATUS    RESTARTS       AGE
    ovnkube-master-99x8j          4/6     Running   7              25d
    ovnkube-master-qdfvv          6/6     Running   14 (15d ago)   25d
    ovnkube-master-w28mh          6/6     Running   7 (15d ago)    25d
    ovnkube-node-5xlxr            5/5     Running   5              34d
    ovnkube-node-6nkm5            5/5     Running   5              34d
    ovnkube-node-dpu-host-45crl   3/3     Running   50             34d
    ovnkube-node-dpu-host-r8tlj   3/3     Running   30             28d
    ovnkube-node-f2x2q            5/5     Running   0              34d
    ovnkube-node-j6w6t            5/5     Running   5              34d
    ovnkube-node-qtc6f            5/5     Running   0              34d
    
    [egallen@bastion-tenant ~]$ oc get pods -n openshift-ovn-kubernetes -o wide
    NAME                          READY   STATUS    RESTARTS       AGE   IP                NODE                     NOMINATED NODE   READINESS GATES
    ovnkube-master-99x8j          4/6     Running   7              25d   192.168.111.122   tenant-master-2          <none>           <none>
    ovnkube-master-qdfvv          6/6     Running   14 (15d ago)   25d   192.168.111.121   tenant-master-1          <none>           <none>
    ovnkube-master-w28mh          6/6     Running   7 (15d ago)    25d   192.168.111.120   tenant-master-0          <none>           <none>
    ovnkube-node-5xlxr            5/5     Running   5              34d   192.168.111.121   tenant-master-1          <none>           <none>
    ovnkube-node-6nkm5            5/5     Running   5              34d   192.168.111.122   tenant-master-2          <none>           <none>
    ovnkube-node-dpu-host-45crl   3/3     Running   50             34d   192.168.111.113   x86-worker-advnetlab13   <none>           <none>
    ovnkube-node-dpu-host-r8tlj   3/3     Running   30             28d   192.168.111.114   x86-worker-advnetlab14   <none>           <none>
    ovnkube-node-f2x2q            5/5     Running   0              34d   192.168.111.123   tenant-worker-0          <none>           <none>
    ovnkube-node-j6w6t            5/5     Running   5              34d   192.168.111.120   tenant-master-0          <none>           <none>
    ovnkube-node-qtc6f            5/5     Running   0              34d   192.168.111.124   tenant-worker-1          <none>           <none>

    Infrastructure cluster: The DaemonSet and the nodes have the same dpu-worker label. The ovnkube-node host network pods will be scheduled on the BlueField-2:

    [egallen@bastion-infrastructure ~]$ oc get ds -n default --show-labels
    NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                     AGE   LABELS
    ovnkube-node   2         2         2       2            2           beta.kubernetes.io/os=linux,node-role.kubernetes.io/dpu-worker=   65d   <none>
    
    [egallen@bastion-infrastructure ~]$ oc get nodes --show-labels | grep dpu-worker
    bf-13      Ready    dpu-worker,worker   71d   v1.22.1+6859754   beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=bf-13,kubernetes.io/os=linux,network.operator.openshift.io/dpu=,node-role.kubernetes.io/dpu-worker=,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos
    bf-14      Ready    dpu-worker,worker   74d   v1.22.1+6859754   beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=bf-14,kubernetes.io/os=linux,network.operator.openshift.io/dpu=,node-role.kubernetes.io/dpu-worker=,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos

    Infrastructure cluster: We see the ovnkube-node DaemonSet running on the two BlueField-2 infrastructure cluster:

    [egallen@bastion-infrastructure ~]$ oc get pods -n default 
    NAME                 READY   STATUS    RESTARTS         AGE
    ovnkube-node-hshxs   2/2     Running   13 (6d13h ago)   25d
    ovnkube-node-mng24   2/2     Running   17 (42h ago)     25d
    
    
    [egallen@bastion-infrastructure ~]$ oc get pods -n default -o wide
    NAME                 READY   STATUS    RESTARTS         AGE   IP               NODE    NOMINATED NODE   READINESS GATES
    ovnkube-node-hshxs   2/2     Running   13 (6d13h ago)   25d   192.168.111.28   bf-14   <none>           <none>
    ovnkube-node-mng24   2/2     Running   17 (42h ago)     25d   192.168.111.29   bf-13   <none>           <none>

    Infrastructure cluster: We get the IP address of the BlueField bf-14 to ssh on it (192.168.111.28):

    [egallen@bastion-infrastructure ~]$ oc get nodes -o wide
    NAME       STATUS   ROLES               AGE   VERSION           INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                       CONTAINER-RUNTIME
    bf-13      Ready    dpu-worker,worker   71d   v1.22.1+6859754   192.168.111.29   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.40.1.el8_4.test.aarch64   cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    bf-14      Ready    dpu-worker,worker   74d   v1.22.1+6859754   192.168.111.28   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.40.1.el8_4.test.aarch64   cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    master-0   Ready    master              75d   v1.22.1+6859754   192.168.111.20   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.30.1.el8_4.aarch64        cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    master-1   Ready    master              75d   v1.22.1+6859754   192.168.111.21   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.30.1.el8_4.aarch64        cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    master-2   Ready    master              75d   v1.22.1+6859754   192.168.111.22   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.30.1.el8_4.aarch64        cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    worker-0   Ready    worker              70d   v1.22.1+6859754   192.168.111.23   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.30.1.el8_4.aarch64        cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8
    worker-1   Ready    worker              70d   v1.22.1+6859754   192.168.111.24   <none>        Red Hat Enterprise Linux CoreOS 410.84.202201071203-0 (Ootpa)   4.18.0-305.30.1.el8_4.aarch64        cri-o://1.23.0-98.rhaos4.10.git9b7f5ae.el8

    Infrastructure cluster: We can ssh to one BlueField-2 of the cluster to check the configuration:

    [egallen@bastion-infrastructure ~]$ ssh core@192.168.111.28 
    Red Hat Enterprise Linux CoreOS 410.84.202201071203-0
      Part of OpenShift 4.10, RHCOS is a Kubernetes native operating system
      managed by the Machine Config Operator (`clusteroperator/machine-config`).
    
    WARNING: Direct SSH access to machines is not recommended; instead,
    make configuration changes via `machineconfig` objects:
      https://docs.openshift.com/container-platform/4.10/architecture/architecture-rhcos.html
    
    ---
    Last login: Wed Mar 30 07:50:50 2022 from 192.168.111.1
    [core@bf-14 ~]$ 
    

    On one BlueField: We are running OpenShift 4.10 on the Arm cores:

    [core@bf-14 ~]$ cat /etc/redhat-release 
    Red Hat Enterprise Linux CoreOS release 4.10
    
    [core@bf-14 ~]$ uname -m
    aarch64
    

    On one BlueField: The BlueField-2 has 8 x Armv8 A72 cores (64-bit):

    [core@bf-13 ~]$ lscpu
    Architecture:        aarch64
    Byte Order:          Little Endian
    CPU(s):              8
    On-line CPU(s) list: 0-7
    Thread(s) per core:  1
    Core(s) per socket:  8
    Socket(s):           1
    NUMA node(s):        1
    Vendor ID:           ARM
    Model:               0
    Model name:          Cortex-A72
    Stepping:            r1p0
    BogoMIPS:            400.00
    L1d cache:           32K
    L1i cache:           48K
    L2 cache:            1024K
    L3 cache:            6144K
    NUMA node0 CPU(s):   0-7
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
    
    

    On one BlueField: dmidecode is a tool for dumping a computer's DMI (also called SMBIOS). The tool is exporting a table of contents. Running on the Jetson gives you an error: “No SMBIOS nor DMI entry point found, sorry.” The command is working on BlueField-2; you can get SMBIOS data from sysfs:

    [core@bf-14 ~]$ sudo dmidecode | grep -A12 "BIOS Information"
    BIOS Information
    	Vendor: https://www.mellanox.com
    	Version: BlueField:3.7.0-20-g98daf29
    	Release Date: Jun 26 2021
    	ROM Size: 4 MB
    	Characteristics:
    		PCI is supported
    		BIOS is upgradeable
    		Selectable boot is supported
    		Serial services are supported (int 14h)
    		ACPI is supported
    		UEFI is supported
    	BIOS Revision: 3.0
    

    On one BlueField: We can get the SOC type with the lshw command:

    [core@bf-14 ~]$ lshw -C system
    bf-14                       
        description: Computer
        product: BlueField SoC (Unspecified System SKU)
        vendor: https://www.mellanox.com
        version: 1.0.0
        serial: Unspecified System Serial Number
        width: 64 bits
        capabilities: smbios-3.1.1 dmi-3.1.1 smp
        configuration: boot=normal family=BlueField sku=Unspecified System SKU uuid=888eecb3-cb1e-40c0-a925-562a7c62ed92
      *-pnp00:00
           product: PnP device PNP0c02
           physical id: 1
           capabilities: pnp
           configuration: driver=system
    
    

    On one BlueField: We have 16GB of RAM on this BlueField-2:

    [core@bf-14 ~]$ free -h
                  total        used        free      shared  buff/cache   available
    Mem:           15Gi       2.3Gi       6.9Gi       193Mi       6.7Gi        10Gi
    Swap:            0B          0B          0B
    

    Infrastructure cluster: I can get the OpenShift console URL (Figure 5) with an oc command:

    [egallen@bastion-infrastructure ~]$ oc whoami --show-console
    https://console-openshift-console.apps.bf2cluster.dev.metalkube.org
    
    [egallen@bastion-infrastructure ~]$ host console-openshift-console.apps.bf2cluster.dev.metalkube.org
    console-openshift-console.apps.bf2cluster.dev.metalkube.org has address 192.168.111.4
    
    Nodes in the OpenShift console.
    Figure 5: Getting information about nodes in the cluster.

    Tenant cluster: In the testpod1 we are running the iperf server:

    [root@testpod-1 /]# taskset -c 6 iperf3 -s                                      
    -----------------------------------------------------------                     
    Server listening on 5201                                                        
    -----------------------------------------------------------                     
    Accepted connection from 10.130.4.132, port 59326                               
    [  5] local 10.129.5.163 port 5201 connected to 10.130.4.132 port 59328         
    [ ID] Interval           Transfer     Bitrate                                   
    [  5]   0.00-1.00   sec  2.35 GBytes  20.2 Gbits/sec                            
    [  5]   1.00-2.00   sec  2.46 GBytes  21.2 Gbits/sec                            
    [  5]   2.00-3.00   sec  2.42 GBytes  20.8 Gbits/sec                            
    [  5]   3.00-4.00   sec  2.24 GBytes  19.2 Gbits/sec                            
    [  5]   4.00-5.00   sec  2.39 GBytes  20.5 Gbits/sec                            
    [  5]   5.00-5.00   sec   384 KBytes  21.2 Gbits/sec                            
    - - - - - - - - - - - - - - - - - - - - - - - - -                               
    [ ID] Interval           Transfer     Bitrate                                   
    [  5]   0.00-5.00   sec  11.9 GBytes  20.4 Gbits/sec                  receiver
    -----------------------------------------------------------                     
    Server listening on 5201                                                        
    -----------------------------------------------------------
                                                                                    
    

    Tenant cluster: In the testpod2, we can get 20-21.2Gbps of throughput with TC Flower OVS hardware offloading instead of 3Gbps of traffic throughput without offloading:

    [root@testpod-2 /]# taskset -c 6 iperf3 -c 10.129.5.163 -t 5
    Connecting to host 10.129.5.163, port 5201
    [  5] local 10.130.4.132 port 59328 connected to 10.129.5.163 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec  2.35 GBytes  20.2 Gbits/sec   17   1.14 MBytes
    [  5]   1.00-2.00   sec  2.46 GBytes  21.2 Gbits/sec    0   1.41 MBytes
    [  5]   2.00-3.00   sec  2.42 GBytes  20.8 Gbits/sec  637   1.01 MBytes
    [  5]   3.00-4.00   sec  2.24 GBytes  19.2 Gbits/sec    0   1.80 MBytes
    [  5]   4.00-5.00   sec  2.39 GBytes  20.6 Gbits/sec    2   1.80 MBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-5.00   sec  11.9 GBytes  20.4 Gbits/sec  656             sender
    [  5]   0.00-5.00   sec  11.9 GBytes  20.4 Gbits/sec                  receiver
    
    iperf Done.
    

    Conclusion

    This example demonstrates the benefits of composable compute architectures that include hardware-level security isolation and capabilities to offload the entire subsystem, such as networking, to the DPU hardware. This enables clean architectural compartmentalization along the same boundaries as the corresponding software services, as well as freeing up working node x83 cores so they can run more applications.

    Unlike SmartNICs that have been niche products with proprietary software stacks, DPUs running Linux, Kubernetes, and open source software stacks offer network function portability. And with Red Hat OpenShift, we are offering long-term support for this stack.

    As DPUs gain more compute power, additional capabilities, and increased popularity in datacenters around the world, Red Hat and NVIDIA plan to continue work on enabling the offloading of additional software functions to DPUs.

    Last updated: May 5, 2022

    Related Posts

    • Sensitive information detection using the NVIDIA Morpheus AI framework

    • Speeding up Open vSwitch with partial hardware offloading

    • Tracing packets inside Open vSwitch

    • Automatic load balancing for PMD threads in Open vSwitch with DPDK

    • Performance improvements in OVN: Past and future

    Recent Posts

    • Container starting and termination order in a pod

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    What’s up next?

    Learn how to deploy a full-stack JavaScript application in an OpenShift cluster with this hands-on lab. Starting from source code, you'll take an application that runs locally and deploy it in the Developer Sandbox for Red Hat OpenShift.

    Get started
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue