Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Open vSwitch-DPDK: How Much Hugepage Memory?

March 16, 2018
Kevin Traynor
Related topics:
LinuxKubernetes
Related products:
Red Hat Enterprise Linux

Share:

    Introduction

    In order to maximize performance of the Open vSwitch DPDK datapath, it pre-allocates hugepage memory. As a user you are responsible for telling Open vSwitch how much hugepage memory to pre-allocate. The question of exactly what value to use often arises. The answer is, it depends.

    There is no simple answer as it depends on things like the MTU size of the ports, the MTU differences between ports, and whether those ports are on the same NUMA node. Just to complicate things a bit more, there are multiple overheads, and alignment and rounding need to be accounted for at various places in OVS-DPDK. Everything clear? OK, you can stop reading then!
    However, if not, read on.

    This blog will help you decide how much hugepage memory to pre-allocate with OVS-DPDK. It's not useful to fill the blog with the many lines of nested C code macros to show the exact calculation for every scenario. Instead we will give some guidelines and examples. This blog is relevant for OVS versions 2.6, 2.7, 2.8, and 2.9.

    What is the Memory Used For?

    The memory we are discussing is primarily used for buffers that represent packets in OVS-DPDK. One or more groups of these buffers are pre-allocated, and any time a new packet arrives on an OVS-DPDK port, one of the buffers is used. An OVS-DPDK port may be the type dpdk for physical NICs, or dpdkvhostuser, or dpdkvhostuserclient for virtual NICs. They will only use buffers that are on the same NUMA node as that which the port is associated with.

    Mempool Sharing

    A group of same size buffers are stored in a ring structure and referred to as a mempool. A mempool can be used for a single port, or it can be shared among multiple ports, on the condition that after some rounding, the ports have matching buffer sizes and are associated with the same NUMA node.

    For example, a physical dpdk port on NUMA 0 with an MTU of 1500 bytes can share a mempool with:

    A virtual vhost port on NUMA 0 with an MTU of 1500 Bytes
    A physical dpdk port on NUMA 0 with an MTU of 1800 Bytes

    However it cannot share a mempool with:

    A virtual vhost port on NUMA 0 with an MTU of 9000 Bytes
    A physical dpdk port on NUMA 1 with an MTU of 1500 Bytes

    Buffer Sizes

    From a user perspective, you just requested an MTU for a port, but that's where the calculations start in OVS-DPDK. Metadata integration with other OVS packet types and alignment rounding all need to be factored in, and are calculated in the code. It will vary according to the MTU and rounding, but it's safe to assume that the overhead will be somewhere between 1000 bytes and 2000 bytes per buffer.

    Let's look at some common examples.

    In a case where you set the MTU of a port to be 1500 bytes:

    Total size per buffer = 3008 Bytes

    In another example where you set the MTU to 9000 bytes:

    Total size per buffer = 10176 Bytes

    That's the size of each individual buffer. The question then is, how many buffers are there in a mempool?

    Mempool Sizes

    We mentioned earlier that mempools can be shared between multiple ports that have similar MTUs and the same NUMA association, and this is not uncommon. To cater for multiple ports using a mempool, initially OVS-DPDK requests 256K buffers in each mempool. It seems like a lot, but we have to consider that for any of these ports the buffers may be in-flight processing, or they could be in-use associated with multiple physical NIC queues. In OVS 2.9 they can also be in software TX queues.

    If 256K buffers are not available, the requested amount is halved up until 16K. If at 16K the mempool can still not be created, an error is reported. Depending on the context of setting the MTU, this either means that the port cannot be correctly created, or that an MTU change failed.

    Let's combine the information about how many buffers are requested for the mempool with the total size of each buffer (from the 1500 Byte and 9000 Byte examples above).

    For an MTU of 1500 bytes:

    Size of requested mempool = 3008 Bytes * 256K
    Size of requested mempool = 788 MBytes

    For an MTU of 9000 bytes:

    Size of requested mempool = 10176 Bytes * 256K
    Size of requested mempool = 2668 MBytes

    Now let's consider a case where the ports with MTUs of 1500 bytes and 9000 bytes co-exist on NUMA node 0. The MTUs are too different to allow sharing of a mempool, so there will be a mempool for each.

    Initially the OVS-DPDK will request:

    Size of mempools = 788 MBytes + 2668 MBytes
    Size of mempools = 3456 MBytes

    The image illustrates the association where there are two physical ports and two virtual ports sharing these MTUs:

    Setting Hugepage Memory

    From the example above, we know we would like to have 3456 MB available on NUMA node 0 to allow for having ports with MTUs of 1500 bytes and 9000 bytes. Let's set it!
    Typically it is set in chunks of 1024 MB, so pre-allocate 4096 MB on NUMA node 0. Strictly speaking, that satisfies the requirement for this example.  However, as noted above, ports will only use mempools from the same NUMA node as which they are associated with, so it's good practice to also pre-allocate on other NUMA nodes.

    # ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=4096,4096

    You can find more information about OVS-DPDK parameters (including dpdk-socket-mem and NUMA) in this blog post.

    Conclusion

    It's difficult to give you a simple calculation for the memory required in order to be pre-allocated for OVS-DPDK. Port MTUs and rounding tend to make this variable. As a guideline, using the examples above should enable a calculation that provides a good approximation. Oh, and the good news is that if you get it wrong, OVS-DPDK will retry using lower amounts of memory.
    So there's a bit of flexibility built in!

    Last updated: September 18, 2018

    Recent Posts

    • Create and enrich ServiceNow ITSM tickets with Ansible Automation Platform

    • Expand Model-as-a-Service for secure enterprise AI

    • OpenShift LACP bonding performance expectations

    • Build container images in CI/CD with Tekton and Buildpacks

    • How to deploy OpenShift AI & Service Mesh 3 on one cluster

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue