Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Troubleshooting Open vSwitch DPDK PMD Thread Core Affinity

June 20, 2018
Eelco Chaudron
Related topics:
Linux
Related products:
Red Hat Enterprise Linux

Share:

    The most common problem when people are trying to deploy an Open vSwitch with Data Plane Development Kit (OvS-DPDK) solution is that the performance is not as expected. For example, they are losing packets. This is where our journey for this series of blogs will start.

    This first blog is about Poll Mode Driver (PMD) thread core affinity. It covers how to configure thread affinity and how to verify that it’s set up correctly. This includes making sure no other threads are using the CPU cores.

    Dedicate CPU Cores to the PMD Threads

    PMD threads are the threads that handle the receiving and processing of packets from the assigned receive queues. They do this in a tight loop, and anything interrupting these threads can cause packets to be dropped. That is why these threads must run on a dedicated CPU core; that is, no other threads in the system should run on this core. This is also true for various Linux kernel tasks.

    Let’s assume you would like to use CPU cores 1 and 15 (a single hyper-thread pair) for your PMD threads. This will convert into a pmd-cpu-mask mask of 0x8002.

    To manually accomplish the isolation you have to do the following:

    • Use the Linux kernel command line option isolcpus to isolate the PMD cores from the general SMP balancing and scheduling algorithms. For the example above, you would use the following: isolcpus=1,15. Please note that the isolcpus= parameter is deprecated in favor of cpusets. For more information check the kernel documentation.
    • Reducing the number of clock tick interrupts can be done with the combined nohz=on nohz_full=1,15 command-line options. This reduces the times the PMD threads get interrupted for servicing timer interrupts. More details on this subject can be found here: NO_HZ.txt
    • For the above to work correctly we need another command-line option, rcu_nocbs=1,15, or else the kernel will still interrupt the thread; details are in the same document: NO_HZ.txt.

    NOTE: For the above kernel options you might need to add additional cores that also need isolation. For example, cores assigned to one or more virtual machines and the cores configured by the dpdk-lcore-mask.

    To make all of the above more convenient you could use a tuned profile called cpu-partitioning for this. There is a somewhat older blog on tuned that might be helpful. However, in short, this is how you configure it:

    # systemctl enable tuned
    # systemctl start tuned
    # echo isolated_cores=1,15 >> /etc/tuned/cpu-partitioning-variables.conf
    # echo no_balance_cores=1,15 >> /etc/tuned/cpu-partitioning-variables.conf
    # tuned-adm profile cpu-partitioning
    # reboot

    NOTE: You still need to set up isolcpus manually as indicated above to achieve 0-loss performance.

    Verify CPU Assignments

    Command-line Options

    First, check the Linux command-line options to see if they are configured as expected:

    # cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 \
    root=/dev/mapper/rhel_wsfd--netdev64-root ro crashkernel=auto \
    rd.lvm.lv=rhel_wsfd-netdev64/root rd.lvm.lv=rhel_wsfd-netdev64/swap \
    console=ttyS1,115200 iommu=pt intel_iommu=on \
    default_hugepagesz=1G hugepagesz=1G hugepages=32 \
    isolcpus=1,2,3,4,5,6,15,16,17,18,19,20 skew_tick=1 \
    nohz=on nohz_full=1,2,3,4,5,6,15,16,17,18,19,20 \
    rcu_nocbs=1,2,3,4,5,6,15,16,17,18,19,20
    tuned.non_isolcpus=0fe07f81 intel_pstate=disable nosoftlockup
    

    PMD Thread Affinity

    Second, make sure the PMD threads are/will be running on the correct threads. In this example you see they are assigned to the wrong CPUs, 11 and 27:

    # pidstat -t -p `pidof ovs-vswitchd` 1 | grep -E pmd\|%CPU
    06:41:21      UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
    06:41:22      995         -      1316  100.00    0.00    0.00  100.00    27  |__pmd33
    06:41:22      995         -      1317  100.00    0.00    0.00  100.00    11  |__pmd32
    06:41:22      UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
    06:41:23      995         -      1316  100.00    0.00    0.00  100.00    27  |__pmd33
    06:41:23      995         -      1317  100.00    0.00    0.00  100.00    11  |__pmd32

    In this case, it’s due to a known bug in tuned that moves away processes running on the isolated cores prior to its initialization. Running sytemctl restart openvswitch will solve this specific issue:

    # systemctl restart openvswitch
    # pidstat -t -p `pidof ovs-vswitchd` 1 | grep -E pmd\|%CPU
    06:44:01      UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
    06:44:02      995         -      2774  100.00    0.00    0.00  100.00     1  |__pmd32
    06:44:02      995         -      2775  100.00    0.00    0.00  100.00    15  |__pmd33
    06:44:02      UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
    06:44:03      995         -      2774  100.00    0.00    0.00  100.00     1  |__pmd32
    06:44:03      995         -      2775  100.00    0.00    0.00  100.00    15  |__pmd33

    NOTE: Until the tuned issue is fixed, you always have to restart OVS after a tuned restart to get the correct CPU assignments!

    To be 100% sure the Linux kernel is not scheduling your PMD thread on another core use the taskset command:

    # taskset -pc 2774
    pid 2774's current affinity list: 1
    # taskset -pc 2775
    pid 2775's current affinity list: 15

    Other Threads Using the PMD Cores

    Finally, make sure that no other threads are scheduled on the PMD cores. The following command will give the CPU affinity for all running userspace threads:

    find -L /proc/[0-9]*/exe ! -type l | cut -d / -f3 | \
      xargs -l -i sh -c 'ps -p {} -o comm=; taskset -acp {}'

    Below is a partial example output. Here you can see that my bash process is using the PMD reserved CPU cores:

    ...
    agetty
    pid 1443's current affinity list: 0,7-14,21-27
    bash
    pid 14863's current affinity list: 0-15
    systemd
    pid 1's current affinity list: 0,7-14,21-27
    ovs-vswitchd
    pid 3777's current affinity list: 2
    pid 3778's current affinity list: 0,7-14,21-27
    pid 3780's current affinity list: 2
    pid 3781's current affinity list: 16
    pid 3782's current affinity list: 2
    pid 3785's current affinity list: 2
    pid 3786's current affinity list: 2
    pid 3815's current affinity list: 1
    pid 3816's current affinity list: 15
    pid 3817's current affinity list: 2
    pid 3818's current affinity list: 2
    pid 3819's current affinity list: 2
    pid 3820's current affinity list: 2
    ...

    NOTE: You could also use a tool called Tuna, to list all processes running on a specific core.

    If you verify all the above, you will prevent other threads and the kernel from interfering with the PMD threads. Don’t forget to re-check the above if you make any major changes to your environment.

     

    Last updated: March 23, 2022

    Recent Posts

    • Build container images in CI/CD with Tekton and Buildpacks

    • How to deploy OpenShift AI & Service Mesh 3 on one cluster

    • JVM tuning for Red Hat Data Grid on Red Hat OpenShift 4

    • Exploring Llama Stack with Python: Tool calling and agents

    • Enhance data security in OpenShift Data Foundation

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue