Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Debugging vHost user TX contention in Open vSwitch

May 29, 2020
David Marchand
Related topics:
DevOpsLinux

Share:

    It isn't always easy to understand how Open vSwitch (OVS) cycles are spent, especially because various parameters and configuration options can affect how OVS behaves. Members of the Open vSwitch community are actively working to understand what causes packets drops in Open vSwitch. Efforts so far have included adding a custom statistic for vHost TX retries, tracking vHost TX contention, and adding a coverage counter to count vHost IRQs. We are particularly interested in the user space datapath that uses the Data Plane Development Kit (DPDK) for fast I/O.

    Adding these statistics is an ongoing effort, and we won't cover all of the corners. In some cases, the statistics leave doubts about what is causing a behavior.

    In this article, I will introduce a new counter we've added to learn more about contention in the vHost transmission path. I'll also show you how to use the new counter with perf, and I'll discuss what's next for our ongoing efforts.

    The test environment for reproducing contention

    In this section, we'll set up a test environment to reproduce contention in the vHost transmission path. Our reference system is running Red Hat Enterprise Linux (RHEL) 7.7 with openvswitch2.11-2.11.0-35.el7fdp.x86_64. You can skip this section if your environment is set up and running already.

    Configuring OVS

    Assuming you have Red Hat Enterprise Linux 7.7 already running in your system, you can configure OVS with a single bridge that has two plugged-in physical ports and two vhost-user-client ports.

    The physical ports are connected to a TRex Realistic Traffic Generator. The traffic generator will send a unidirectional flow of packets to the bridge, passed to the vhost-user-clients ports. We've crafted these packets to send traffic to both queues of the first physical port on the OVS side. The vhost-user-clients ports are connected to a virtual machine (VM) that sends the packets back using testpmd in io forward mode.

    If you need more details about setting up a TRex traffic generator, configuring a host running OVS, or other aspects of this setup, see Eelco Chaudron's introduction to Automated Open vSwitch PVP testing.

    Configuring the bridge and host

    The setup in the ASCII diagram here differs slightly from the Physical interface to Virtual interface back to Physical interface (PVP) setup shown in the linked article above.

    +------+   +-----+   +---------+
    |      |   |     |   |         |
    |     0+---+1   4+---+0        |
    | tgen |   | ovs |   | testpmd |
    |     1+---+2   5+---+1        |
    |      |   |     |   |         |
    +------+   +-----+   +---------+
    

    Configure the bridge on the host as follows:

    # ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true
    # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x00008002
    # ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
    # ovs-vsctl add-port br0 dpdk0 -- \
        set Interface dpdk0 type=dpdk -- \
        set Interface dpdk0 options:dpdk-devargs=0000:01:00.0 -- \
        set Interface dpdk0 ofport_request=1 -- \
        set Interface dpdk0 options:n_rxq=2
    # ovs-vsctl add-port br0 dpdk1 -- \
        set Interface dpdk1 type=dpdk -- \
        set Interface dpdk1 options:dpdk-devargs=0000:01:00.1 -- \
        set Interface dpdk1 ofport_request=2 -- \
        set Interface dpdk1 options:n_rxq=2
    # ovs-vsctl add-port br0 vhost0 -- \
        set Interface vhost0 type=dpdkvhostuserclient -- \
        set Interface vhost0 options:vhost-server-path="/tmp/vhost-sock0" -- \
        set Interface vhost0 ofport_request=4
    # ovs-vsctl add-port br0 vhost1 -- \
        set Interface vhost1 type=dpdkvhostuserclient -- \
        set Interface vhost1 options:vhost-server-path="/tmp/vhost-sock1" -- \
        set Interface vhost1 ofport_request=5
    

    Check the polling configuration:

    # ovs-appctl dpif-netdev/pmd-rxq-show
    pmd thread numa_id 0 core_id 1:
      isolated : false
      port: dpdk0             queue-id:  0  pmd usage: NOT AVAIL
      port: dpdk1             queue-id:  1  pmd usage: NOT AVAIL
      port: vhost0            queue-id:  0  pmd usage: NOT AVAIL
    pmd thread numa_id 0 core_id 15:
      isolated : false
      port: dpdk0             queue-id:  1  pmd usage: NOT AVAIL
      port: dpdk1             queue-id:  0  pmd usage: NOT AVAIL
      port: vhost1            queue-id:  0  pmd usage: NOT AVAIL
    

    We could let this OVS bridge with a NORMAL action, in which case it would behave like a standard switch learning Media Access Control (MAC) addresses on its ports. To simplify the setup, let's just write a few OpenFlow rules for a basic mapping:

    • Receiving on the physical port dpdk0 pushes packets to vhost0.
    • Receiving on the virtual port vhost0 pushes packets to dpdk0.
    • Receiving on the physical port dpdk1 pushes packets to vhost1.
    • Receiving on the virtual port vhost1 pushes packets to dpdk1.

    Here's that mapping:

    # ovs-ofctl del-flows br0
    # ovs-ofctl add-flow br0 in_port=1,actions=4
    # ovs-ofctl add-flow br0 in_port=4,actions=1
    # ovs-ofctl add-flow br0 in_port=2,actions=5
    # ovs-ofctl add-flow br0 in_port=5,actions=2
    

    That completes the test environment.

    Catching vHost TX contention

    Now let's take a look at the new coverage counter for OVS:

    # ovs-appctl coverage/show |grep vhost
    vhost_tx_contention      39082.8/sec 11553.017/sec      192.5503/sec   total: 758359
    

    Adding a perf probe

    As it is, the counter leaves the question of which cores are impacted by contention. We can use perf to catch more information without stopping OVS. Just add a probe in the branch where the contention occurs:

    # perf probe -x $(which ovs-vswitchd) 'netdev_dpdk_vhost_tx_lock=__netdev_dpdk_vhost_send:22 netdev->name:string qid'
    Added new event:
      probe_ovs:netdev_dpdk_vhost_tx_lock (on __netdev_dpdk_vhost_send:22 in /usr/sbin/ovs-vswitchd with name=netdev->name:string qid)
    

    Now you can use the counter in all of your perf tools.

    Using the coverage counter in perf

    Here, we ask perf to record a specific event:

    # perf record -e probe_ovs:netdev_dpdk_vhost_tx_lock -aR sleep 1
    [ perf record: Woken up 15 times to write data ]
    [ perf record: Captured and wrote 3.938 MB perf.data (44059 samples) ]
    

    We can also make a report of this perf session:

    # perf report -F +pid --stdio
    # To display the perf.data header info, please use --header/--header-only options.
    #
    #
    # Total Lost Samples: 0
    #
    # Samples: 44K of event 'probe_ovs:netdev_dpdk_vhost_tx_lock'
    # Event count (approx.): 44059
    #
    # Overhead      Pid:Command  Trace output
    # ........  ...............  ..................................
    #
        61.30%    33003:pmd60    (55ef4abe5494) name="vhost0" qid=0
        38.70%    33006:pmd61    (55ef4abe5494) name="vhost0" qid=0
    
    #
    # (Tip: For a higher level overview, try: perf report --sort comm,dso)
    #
    

    The new coverage counter makes interpreting contention easier. We can see that the contention happened between pmd60 (on core 1, by looking at the OVS logs) and pmd61 (on core 15). Both pmd threads are trying to send packets on the vhost0 port queue zero.

    Conclusion

    Using perf to debug contention is interesting, and it worked in this case because we were trying to catch events on an error or slow path. But perf involves context switches that have a visible effect on performance. We can't use it without accounting for the performance impact.

    Even if it's fine for developers to put in a probe by reading the code sources, support or operations teams will prefer higher-level tools or traces. The DPDK community has started a workgroup to set traces with a minimal impact at key places in the DPDK infrastructure code. We are still far from something as rich as perf, but this is likely to be a focus for part of the community for the next year.

    Last updated: June 26, 2020

    Recent Posts

    • How to run AI models in cloud development environments

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue