Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Measuring and comparing Open vSwitch performance

June 5, 2017
Eelco Chaudron
Related topics:
Linux
Related products:
Red Hat Enterprise Linux

Share:

    Introduction

    There are infinite ways to test Virtual Switches, all tailored to expose (or hide) a specific characteristic. The goal of our test is to measure performance under stress and be able to compare versions, which may or may not have hardware offload.

    We will run the tests using the Physical to Virtual back to Physical topology. This configuration is also known as the PVP setup. The traffic will flow from a physical port to a virtual port on the Virtual Machine (VM), and then back to the physical port. The picture below shows the flow through Open vSwitch-DPDK, which we will use as an example throughout this blog post:

    Measuring and comparing Open vSwitch performance

    Traffic pattern and OpenFlow rules

    The used traffic pattern and the configured OpenFlow rules can greatly influence the performance. For example:

    • A traffic pattern that is not fairly distributed to the Multiqueue Network Interface Card's queues. Like when only the L2 fields (source and destination MAC addresses) in the traffic stream change.
    • A single IPv4 /16 OpenFlow rule vs. 64K /32 rules.

    So, depending on what aspects you want to test you configure traffic patterns and a set of OpenFlow rules. For our case, we just want to verify a bulk of traffic where each traffic stream has its own OpenFlow rule. Actually, we will have two OpenFlow rules for each traffic stream, one going to the Virtual Machine (VM) and one going back.

    Due to some limitation on our traffic generator, we decided to only change the source and destination IPv4 addresses. This should not be a problem, as a number of streams we generate are causing a balanced distribution amongst the Multiqueue Network Interface Card's queues.

    The packets we use for the purpose of this blog have an Ethernet II header (no VLAN), IPv4 header followed by an UDP header (with static ports). The size of the UDP data will be such that it fills the physical packet.

    Especially for ASCII art lovers below is a representation of the packet. The fields in bold will change for each flow.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Destination MAC Address                                       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Destination MAC (continue)    | Source MAC Address            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Source MAC Address (continue)                                 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | EtherType                     |
    ===============================IP4===============================
    |Version|  IHL  |Type of Service|          Total Length         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         Identification        |Flags|      Fragment Offset    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  Time to Live |    Protocol   |         Header Checksum       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                       Source Address                          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Destination Address                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Options                    |    Padding    |
    ===============================UDP===============================
    |          Source Port          |       Destination Port        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |            Length             |           Checksum            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |   .... data ....                                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    

    The VM will loopback the packets without modification to avoid wasting CPU cycles on the guest (see below). This will result in a single traffic stream (flowing in both directions), which gets associated with two OpenFlow rules (one for each direction). For the remaining of this blog post we will use the word flow to associate this pattern, i.e. 10 flows mean the traffic generator will create 10 traffic streams, and we configure 20 OpenFlow rules.

    Setting up Red Hat Enterprise Linux

    Here we assume you just finished installing Red Hat Enterprise Linux Server 7.3, including registration. We can now continue by enabling the two repositories we need. We need "Red Hat Enterprise Linux Fast Datapath 7" for Open vSwitch, and "Red Hat Virtualization 4" for Qemu. If you do not have access to these repositories, please contact your Red Hat representative.

    subscription-manager repos --enable=rhel-7-fast-datapath-rpms
    subscription-manager repos --enable=rhel-7-server-rhv-4-mgmt-agent-rpms

    Installing the required packages

    Next, we will install the required packages.

    yum -y clean all
    yum -y update
    yum -y install driverctl git intltool libguestfs libguestfs-tools-c \
                   libosinfo libvirt libvirt-python lshw openvswitch \
                   python-ipaddr python-requests qemu-kvm-rhel

    Tuning the host for Open vSwitch with DPDK

    We need to dedicate some CPUs and huge page memory to the Open vSwitch's DPDK PMD threads and the Virtual Machine that will loopback the traffic. For quick benchmarking, I prefer to do the tuning manually. Alternatively, you can use tuned to do some of this for you.

    As these CPUs should be dedicated to run Open vSwitch and the VM, they should be interrupted as minimal as possible. To accomplish this, we need to disable the timer ticks and general interrupt processing on those CPUs.

    In our test scenario, we will reserve five CPUs for Open vSwitch, and another five for the Virtual Machine. We will use the CPUs numbered 1 to 10.

    To reserve the huge page memory, the CPUs and disabling the timer ticks the following needs to be added to the kernel command line:

    default_hugepagesz=1G hugepagesz=1G hugepages=16
    isolcpus=1-10
    nohz_full=1-10 rcu_nocbs=1-10 rcu_nocb_poll

    In addition, we have to make sure iommu is enabled for the generic vfi-pci driver used by DPDK. As this is also a kernel command line option, we do it here:

    iommu=pt intel_iommu=on

    One way of getting these command line options applied is to edit the "/etc/default/grub" file and execute the grub2-mkconfig -o /boot/grub2/grub.cfg command.

    NOTE: After configuring the command line options, you MUST reboot the host!

    Disable IRQs from happening on the dedicated CPUs by removing them from the smp_affinity mask. In addition, we also disable IRQ re-balancing, and the NMI watchdog interrupts:

    systemctl disable irqbalance
    systemctl stop irqbalance
    echo 0 > /proc/sys/kernel/nmi_watchdog
    for I in /proc/irq/[0-9]*; do
        echo fffff801 > ${I}/smp_affinity >/dev/null 2>&1
    done

    NOTE: With this approach, it needs to be done after every reboot!

    Setting up the Open vSwitch dedicated physical interface

    In the Open vSwitch DPDK configuration the physical interface is under the direct control of DPDK, hence it needs to be removed from the kernel. To do this we first need to figure out the interface's PCI address. An easy way of doing this is using the lshw utility:

    # lshw -c network -businfo
    Bus info          Device      Class          Description
    ========================================================
    pci@0000:01:00.0  em1         network        82599ES 10-Gigabit SFI/SFP+ Network
    pci@0000:01:00.1  em2         network        82599ES 10-Gigabit SFI/SFP+ Network
    pci@0000:05:00.0  p5p1        network        Ethernet Controller XL710 for 40GbE
    pci@0000:05:00.1  p5p2        network        Ethernet Controller XL710 for 40GbE
    pci@0000:07:00.0  em3         network        I350 Gigabit Network Connection
    pci@0000:07:00.1  em4         network        I350 Gigabit Network Connection

    For our performance test, we would like to use the 40GbE interface p5p1. You could use the dpdk-devbind utility to bind the interface to DPDK; however, this configuration will not survive a reboot. The preferred solution is to use driverctl:

    # driverctl -v set-override 0000:05:00.0 vfio-pci
    driverctl: setting driver override for 0000:05:00.0: vfio-pci
    driverctl: loading driver vfio-pci
    driverctl: unbinding previous driver i40e
    driverctl: reprobing driver for 0000:05:00.0
    driverctl: saving driver override for 0000:05:00.0
    
    # lshw -c network -businfo
    Bus info          Device      Class          Description
    ========================================================
    pci@0000:01:00.0  em1         network        82599ES 10-Gigabit SFI/SFP+ Network
    pci@0000:01:00.1  em2         network        82599ES 10-Gigabit SFI/SFP+ Network
    pci@0000:05:00.0              network        Ethernet Controller XL710 for 40GbE
    pci@0000:05:00.1  p5p2        network        Ethernet Controller XL710 for 40GbE
    pci@0000:07:00.0  em3         network        I350 Gigabit Network Connection
    pci@0000:07:00.1  em4         network        I350 Gigabit Network Connection

    NOTE: Make sure iommu is enabled as explained above.

    Disable SELinux

    There is work in progress for Open vSwitch DPDK to play nicely with SELinux, but for now, the easiest way is to disable it.

    sed -i -e 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
    setenforce permissive

    Setting up Open vSwitch

    Start Open vSwitch and enable DPDK

    Start Open vSwitch, and automatically start it after every reboot:

    systemctl enable openvswitch
    systemctl start openvswitch

    Now that it's fired up, let's enable DPDK and assign the PMD cores.

    ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true
    ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=2048
    ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x2
    ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3c
    systemctl restart openvswitch

    Bridge configuration

    For the Physical to Virtual back to Physical(PVP) test, we only need one bridge with two ports. In addition, we will configure our interfaces with 4 receive queues:

    ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
    ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk -- \
      set Interface dpdk0 ofport_request=1
    ovs-vsctl add-port br0 vhost0 -- set Interface vhost0 type=dpdkvhostuser -- \
      set Interface vhost0 ofport_request=2
    ovs-vsctl set interface dpdk0 options:n_rxq=4 \
      other_config:pmd-rxq-affinity="0:2,1:3,2:4,3:5"
    ovs-vsctl set interface vhost0 options:n_rxq=4 \
      other_config:pmd-rxq-affinity="0:2,1:3,2:4,3:5"

    Also, we need to change access rights to the vhost0 socket:

     chmod 777 /var/run/openvswitch/vhost0

    Setting up the Virtual Machine

    We will use virsh to manage the one Virtual Machine we use for looping back the traffic. The Virtual Machine will be an instance of Centos 7.3 which we get using virt-builder.

    Download a Virtual Machine image

    We will download the Virtual Machine qcow2 image using virt-builder, setting the root password to centos:

    mkdir -p /opt/images
    cd /opt/images
    export LIBGUESTFS_BACKEND=direct
    virt-builder centos-7.3 --root-password password:centos -o centos_loopback.qcow2 --format qcow2

    Get the latest version of virt-manager

    Now that we have the disk image, we need to create the Virtual Machine profile. We do this with virt-install, which we will install from its GIT repository:

    cd ~
    git clone https://github.com/virt-manager/virt-manager.git
    cd ~/virt-manager
    python setup.py --no-update-icon-cache install

    Create and configure the Virtual Machine Profile

    First, we need to start and enable libvirtd:

    systemctl enable libvirtd.service
    systemctl start libvirtd.service

    Setup as much as possible with a single call to virt-install:

    virt-install --connect=qemu:///system \
      --network vhostuser,source_type=unix,source_path=/var/run/openvswitch/vhost0,source_mode=client,model=virtio,driver_queues=4 \
      --network network=default \
      --name=centos_loopback \
      --disk path=/opt/images/centos_loopback.qcow2,format=qcow2 \
      --ram 4096 \
      --memorybacking hugepages=on,size=1024,unit=M,nodeset=0 \
      --vcpus=5,cpuset=6,7,8,9,10 \
      --check-cpu \
      --cpu Haswell-noTSX,cell0.id=0,cell0.cpus=0,cell0.memory=4194304 \
      --numatune mode=strict,nodeset=0 \
      --nographics --noautoconsole \
      --import \
      --os-variant=rhel7

    We need to tweak some Virtual Machine profile settings manually, as not all options are available through virt-install. This is related to memory sharing, and pinning of the Virtual Machine to dedicated CPUs. We will do this using virsh edit. Below are the commands used, and the diff of the applied changes:

    virsh shutdown centos_loopback
    virsh edit centos_loopback
    
    diff:
    @@ -18,2 +18,8 @@
     <vcpu placement='static' cpuset='6-10'>5</vcpu>
    + <cputune>
    + <vcpupin vcpu='0' cpuset='6'/>
    + <vcpupin vcpu='1' cpuset='7'/>
    + <vcpupin vcpu='2' cpuset='8'/>
    + <vcpupin vcpu='3' cpuset='9'/>
    + <vcpupin vcpu='4' cpuset='10'/>
    + </cputune>
     <numatune>
    @@ -32,3 +38,3 @@
     <numa>
    - <cell id='0' cpus='0' memory='4194304' unit='KiB'/>
    + <cell id='0' cpus='0' memory='4194304' unit='KiB' memAccess='shared'/>
     </numa></pre>
    

    This small start-up script will be copied to the Virtual Machine for ease of use:

    cat << ENDL > start_testpmd.sh
    depmod -a 
    modprobe uio_pci_generic
    echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    driverctl -v set-override 0000:00:02.0 uio_pci_generic
    (while sleep 1; do echo show port stats 0; done | \
    testpmd -c 0x1f -n 4 --socket-mem 1024,0 -w 0000:00:02.0 -- --burst 64 \
      --disable-hw-vlan -i --rxq=4 --txq=4 --rxd=4096 --txd=1024 \
      --coremask=0x1e --auto-start --port-topology=chained)
    ENDL

    Tweak the virtual machine such that it will have the interfaces named trough network manager, and install the DPDK packages on the next boot.

    LIBGUESTFS_BACKEND=direct virt-customize -d centos_loopback \
      --upload start_testpmd.sh:/root/ \
      --firstboot-command 'nmcli c | grep -o --  "[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{12\}" | xargs -n 1 nmcli c delete uuid' \
      --firstboot-command 'nmcli con add con-name ovs-dpdk ifname eth0 type ethernet ip4 1.1.1.1/24' \
      --firstboot-command 'nmcli con add con-name management ifname eth1 type ethernet' \
      --firstboot-command 'chmod +x /root/start_testpmd.sh' \
      --firstboot-command 'yum install -y https://copr-be.cloud.fedoraproject.org/results/tredaell/dpdk/epel-7-x86_64/00522541-dpdk/dpdk-16.11.1-1.el7.centos.x86_64.rpm' \
      --firstboot-command 'yum install -y https://copr-be.cloud.fedoraproject.org/results/tredaell/dpdk/epel-7-x86_64/00522541-dpdk/dpdk-tools-16.11.1-1.el7.centos.x86_64.rpm' \
      --firstboot-command 'yum install -y driverctl'

    Running the test

    Looping back the traffic

    We will start the virtual machine, and once attached to the console we log in and start the loopback.

    virsh start centos_loopback
    virsh console centos_loopback
    Connected to domain centos_loopback
    Escape character is ^]
    
    CentOS Linux 7 (Core)
    Kernel 3.10.0-514.el7.x86_64 on an x86_64
    
    localhost login: root
    Password:
    Last login: Tue May 23 03:45:40 on ttyS0
    [root@localhost ~]# ./start_testpmd.sh 
       46.684337] Generic UIO driver for PCI 2.3 devices version: 0.01.0
    EAL: Detected 5 lcore(s)
    EAL: Probing VFIO support...
    [   50.563993] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm/pagemap.txt for details.
    EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
    EAL: PCI device 0000:00:02.0 on NUMA socket -1
    EAL:   probe driver: 1af4:1000 net_virtio
    Interactive-mode selected
    previous number of forwarding cores 1 - changed to number of configured cores 4
    Auto-start selected
    USER1: create a new mbuf pool : n=179456, size=2176, socket=0
    Configuring Port 0 (socket 0)
    Port 0: 52:54:00:A0:2A:FC
    Checking link statuses...
    Port 0 Link Up - speed 10000 Mbps - full-duplex
    Done
    Start automatic packet forwarding
    io packet forwarding - ports=1 - cores=4 - streams=4 - NUMA support disabled, MP over anonymous pages disabled
    Logical Core 1 (socket 0) forwards packets on 1 streams:
      RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
    Logical Core 2 (socket 0) forwards packets on 1 streams:
      RX P=0/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
    Logical Core 3 (socket 0) forwards packets on 1 streams:
      RX P=0/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
    Logical Core 4 (socket 0) forwards packets on 1 streams:
      RX P=0/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
    
      io packet forwarding - CRC stripping disabled - packets/burst=64
      nb forwarding cores=4 - nb forwarding ports=1
      RX queues=4 - RX desc=4096 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0 wthresh=0
      TX queues=4 - TX desc=1024 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0 wthresh=0
      TX RS bit threshold=0 - TXQ flags=0xf00
    testpmd> show port stats 0
    
      ######################## NIC statistics for port 0  ########################
      RX-packets: 0          RX-missed: 0          RX-bytes:  0
      RX-errors: 0
      RX-nombuf:  0         
      TX-packets: 0          TX-errors: 0          TX-bytes:  0
    
      Throughput (since last show)
      Rx-pps:            0
      Tx-pps:            0
      ############################################################################
    testpmd>

    The start_testpmd.sh shell script will query the statistics every second, so you can see the packet coming in and going out of the interface.

    Configuring the traffic generator

    We configure the traffic generator to send packets with the required packet size at wire speed. Note that Open vSwitch will probably not be able to handle this traffic volume, but that's the purpose of this test.

    As mentioned earlier we will use UDP packets where only the Source and Destination address are changing. The MAC addresses and UDP ports are static.

    Assuming we need 10 traffic streams, we will use these IP addresses:

    • ip_src=1.0.0.0, ip_dst=2.0.0.0
    • ip_src=1.0.0.1, ip_dst=2.0.0.1
    • ip_src=1.0.0.2, ip_dst=2.0.0.2
    • ip_src=1.0.0.3, ip_dst=2.0.0.3
    • ip_src=1.0.0.4, ip_dst=2.0.0.4
    • ip_src=1.0.0.5, ip_dst=2.0.0.5
    • ip_src=1.0.0.6, ip_dst=2.0.0.6
    • ip_src=1.0.0.7, ip_dst=2.0.0.7
    • ip_src=1.0.0.8, ip_dst=2.0.0.8
    • ip_src=1.0.0.9, ip_dst=2.0.0.9

    Setting up the OpenFlow rules

    Looking at the above 10 traffic streams, we need the following 20 OpenFlow rules (10 in each direction):

    ovs-ofctl del-flows br0
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.0,nw_dst=2.0.0.0,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.1,nw_dst=2.0.0.1,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.2,nw_dst=2.0.0.2,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.3,nw_dst=2.0.0.3,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.4,nw_dst=2.0.0.4,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.5,nw_dst=2.0.0.5,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.6,nw_dst=2.0.0.6,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.7,nw_dst=2.0.0.7,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.8,nw_dst=2.0.0.8,action=2"
    ovs-ofctl add-flow br0 "in_port=1,eth_type(0x800),nw_src=1.0.0.9,nw_dst=2.0.0.9,action=2"
    
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.0,nw_dst=2.0.0.0,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.1,nw_dst=2.0.0.1,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.2,nw_dst=2.0.0.2,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.3,nw_dst=2.0.0.3,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.4,nw_dst=2.0.0.4,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.5,nw_dst=2.0.0.5,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.6,nw_dst=2.0.0.6,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.7,nw_dst=2.0.0.7,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.8,nw_dst=2.0.0.8,action=1"
    ovs-ofctl add-flow br0 "in_port=2,eth_type(0x800),nw_src=1.0.0.9,nw_dst=2.0.0.9,action=1"

    Send the traffic

    Start sending the traffic, and watch the return rate on the traffic generator. If traffic is not returning you could check the traffic hitting the OpenFlow rules:

    # ovs-ofctl dump-flows br0
    NXST_FLOW reply (xid=0x4):
     cookie=0x0, duration=13.670s, table=0, n_packets=21117649, n_bytes=2618588476, idle_age=0, ip,in_port=1,nw_src=1.0.0.0,nw_dst=2.0.0.0 actions=output:2
     cookie=0x0, duration=13.445s, table=0, n_packets=21092896, n_bytes=2615519104, idle_age=0, ip,in_port=1,nw_src=1.0.0.1,nw_dst=2.0.0.1 actions=output:2
     cookie=0x0, duration=13.229s, table=0, n_packets=19727728, n_bytes=2446238272, idle_age=0, ip,in_port=1,nw_src=1.0.0.2,nw_dst=2.0.0.2 actions=output:2
     cookie=0x0, duration=13.011s, table=0, n_packets=19474480, n_bytes=2414835520, idle_age=0, ip,in_port=1,nw_src=1.0.0.3,nw_dst=2.0.0.3 actions=output:2
     cookie=0x0, duration=12.362s, table=0, n_packets=8768808, n_bytes=1087332192, idle_age=0, ip,in_port=1,nw_src=1.0.0.4,nw_dst=2.0.0.4 actions=output:2
     cookie=0x0, duration=12.167s, table=0, n_packets=8749840, n_bytes=1084980160, idle_age=0, ip,in_port=1,nw_src=1.0.0.5,nw_dst=2.0.0.5 actions=output:2
     cookie=0x0, duration=11.976s, table=0, n_packets=19274081, n_bytes=2389986044, idle_age=0, ip,in_port=1,nw_src=1.0.0.6,nw_dst=2.0.0.6 actions=output:2
     cookie=0x0, duration=11.594s, table=0, n_packets=19241248, n_bytes=2385914752, idle_age=0, ip,in_port=1,nw_src=1.0.0.7,nw_dst=2.0.0.7 actions=output:2
     cookie=0x0, duration=11.416s, table=0, n_packets=8678017, n_bytes=1076074108, idle_age=0, ip,in_port=1,nw_src=1.0.0.8,nw_dst=2.0.0.8 actions=output:2
     cookie=0x0, duration=11.275s, table=0, n_packets=8670176, n_bytes=1075101824, idle_age=0, ip,in_port=1,nw_src=1.0.0.9,nw_dst=2.0.0.9 actions=output:2
     cookie=0x0, duration=11.095s, table=0, n_packets=20746433, n_bytes=2572557692, idle_age=0, ip,in_port=2,nw_src=1.0.0.0,nw_dst=2.0.0.0 actions=output:1
     cookie=0x0, duration=10.628s, table=0, n_packets=20664334, n_bytes=2562377416, idle_age=0, ip,in_port=2,nw_src=1.0.0.1,nw_dst=2.0.0.1 actions=output:1
     cookie=0x0, duration=10.570s, table=0, n_packets=17456255, n_bytes=2164575620, idle_age=0, ip,in_port=2,nw_src=1.0.0.2,nw_dst=2.0.0.2 actions=output:1
     cookie=0x0, duration=10.487s, table=0, n_packets=17441918, n_bytes=2162797832, idle_age=0, ip,in_port=2,nw_src=1.0.0.3,nw_dst=2.0.0.3 actions=output:1
     cookie=0x0, duration=10.405s, table=0, n_packets=8557728, n_bytes=1061158272, idle_age=0, ip,in_port=2,nw_src=1.0.0.4,nw_dst=2.0.0.4 actions=output:1
     cookie=0x0, duration=10.344s, table=0, n_packets=8551536, n_bytes=1060390464, idle_age=0, ip,in_port=2,nw_src=1.0.0.5,nw_dst=2.0.0.5 actions=output:1
     cookie=0x0, duration=10.261s, table=0, n_packets=17106209, n_bytes=2121169916, idle_age=0, ip,in_port=2,nw_src=1.0.0.6,nw_dst=2.0.0.6 actions=output:1
     cookie=0x0, duration=10.191s, table=0, n_packets=17074570, n_bytes=2117246680, idle_age=0, ip,in_port=2,nw_src=1.0.0.7,nw_dst=2.0.0.7 actions=output:1
     cookie=0x0, duration=10.158s, table=0, n_packets=8529608, n_bytes=1057671392, idle_age=0, ip,in_port=2,nw_src=1.0.0.8,nw_dst=2.0.0.8 actions=output:1
     cookie=0x0, duration=10.133s, table=0, n_packets=8523789, n_bytes=1056949836, idle_age=0, ip,in_port=2,nw_src=1.0.0.9,nw_dst=2.0.0.9 actions=output:1

    In addition, you can check the testpmd statistics:

    testpmd> show port stats 0
    
      ######################## NIC statistics for port 0  ########################
      RX-packets: 2278543290 RX-missed: 0          RX-bytes:  282539370192
      RX-errors: 0
      RX-nombuf:  0         
      TX-packets: 2241649130 TX-errors: 0          TX-bytes:  277964492120
    
      Throughput (since last show)
      Rx-pps:     14337207
      Tx-pps:     14151789
      ############################################################################

    Results

    Below is a graph showing performance results for various packet sizes and number of traffic flows for the test methodology described in this blog. Note that this is NOT a zero packet loss test, as we are pushing packets at wire speed. The graph also shows the systems CPU usage during these tests. The CPU data was captured using the pidstat and mpstat utilities.

    The following system configuration was used to gather these numbers and create the graph:

    • Dell PowerEdge R730, single socket
    • Intel Xeon E5-2690 v4 @ 2.60GHz
    • 128G of system memory
    • Intel XL710 2x 40G adapter

    If you are thinking of using a PVP test to compare various versions of Open vSwitch, or various Open vSwitch hardware offload solutions, test them in such a way you are comparing apples to apples. To be more specific, on the same hardware configuration (or even better same machine). If you are comparing Open vSwitch DPDK versions, use the same number of PMD threads/interface queues. For testpmd on the Virtual Machine use the same options, i.e. a number of threads, interface queues, and the number of receive/transmit buffer descriptors, etc. etc., for all tests.

    Last updated: February 22, 2024

    Recent Posts

    • Assessing AI for OpenShift operations: Advanced configurations

    • OpenShift Lightspeed: Assessing AI for OpenShift operations

    • OpenShift Data Foundation and HashiCorp Vault securing data

    • Axolotl meets LLM Compressor: Fast, sparse, open

    • What’s new for developers in Red Hat OpenShift 4.19

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue