Dynamic IP address management in Open Virtual Network (OVN): Part Two

In part one of this series, we explored the dynamic IP address management (IPAM) capabilities of Open Virtual Network. We covered the subnet, ipv6_prefix, and exclude_ips options on logical switches. We then saw how these options get applied to logical switch ports whose addresses have been set to the special “dynamic” value.  OVN, a subproject of Open vSwitch, is used for virtual networking in a number of Red Hat products like Red Hat OpenStack Platform, Red Hat Virtualization, and Red Hat OpenShift Container Platform in a future release.

In this part, we’re going to explore some of the oversights and downsides in the feature, how those have been corrected, and what’s in store for OVN in future versions.

Continue reading “Dynamic IP address management in Open Virtual Network (OVN): Part Two”


Troubleshooting FDB table wrapping in Open vSwitch

When most people deploy an Open vSwitch configuration for virtual networking using the NORMAL rule, that is, using L2 learning, they do not think about configuring the size of the Forwarding DataBase (FDB).

When hardware-based switches are used, the FDB size is generally rather large and the large FDB size is a key selling point. However for Open vSwitch, the default FDB value is rather small, for example, in version 2.9 and earlier it is only 2K entries. Starting with version 2.10 the FDB size was increased to 8K entries. Note that for Open vSwitch, each bridge has its own FDB table for which the size is individually configurable.

This blog explains the effects of configuring too small an FDB table, how to identify which bridge is suffering from too small an FDB table, and how to configure the FDB table size appropriately.

Continue reading “Troubleshooting FDB table wrapping in Open vSwitch”


Dynamic IP Address Management in Open Virtual Network (OVN): Part One

Some background

For those unfamiliar, Open Virtual Network (OVN) is a subproject of OpenVswitch (OVS), a performant programmable multi-platform virtual switch. OVN provides the ability to express an overlay network as a series of virtual routers and switches. OVN also provides native methods for setting up Access Control Lists (ACLs), and it functions as an OpenFlow switch, providing services such as DHCP. The components of OVN program OVS on each of the hypervisors in the network. Many of Red Hat’s products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN. Red Hat OpenShift Container Platform will be using OVN soon.

Looking around the internet, it’s pretty easy to find high-quality tutorials on the basics of OVN. However, when it comes to more-advanced topics, it sometimes feels like the amount of information is lacking. In this tutorial, we’ll examine dynamic addressing in OVN. You will learn about IP address management (IPAM) options in OVN and how to apply them.

Continue reading “Dynamic IP Address Management in Open Virtual Network (OVN): Part One”


Debugging Memory Issues with Open vSwitch DPDK


This article is about debugging out-of-memory issues with Open vSwitch with the Data Plane Development Kit (OvS-DPDK). It explains the situations in which you can run out of memory when using OvS-DPDK and it shows the log entries that are produced in those circumstances. It also shows some other log entries and commands for further debugging.

When you finish reading this article, you will be able to identify that you have an out-of-memory issue and you’ll know how to fix it. Spoiler: Usually having some more memory on the relevant NUMA node works. It is based on OvS 2.9.

Continue reading “Debugging Memory Issues with Open vSwitch DPDK”


Non-root Open vSwitch in RHEL

In a few weeks, the Fast Datapath Production channel will update the Open vSwitch version from the 2.7 series to the 2.9 series. This is an important change in more ways than one. A wealth of new features and fixes all related to packet movement will come into play. One that will surely be blamed for all your troubles will be the integration of the `–ovs-user` flag to allow for an unprivileged user to interact with Open vSwitch.

Running as root can solve a lot of pesky problems. Want to write to an arbitrary file? No problem. Want to load kernel modules? Go for it! Want to sniff packets on the wire? Have a packet dump. All of these are great when the person commanding the computer is the rightful owner. But the moment the person in front of the keyboard isn’t the rightful owner, problems occur.

Continue reading “Non-root Open vSwitch in RHEL”


Open vSwitch-DPDK: How Much Hugepage Memory?


In order to maximize performance of the Open vSwitch DPDK datapath, it pre-allocates hugepage memory. As a user you are responsible for telling Open vSwitch how much hugepage memory to pre-allocate. The question of exactly what value to use often arises. The answer is, it depends.

There is no simple answer as it depends on things like the MTU size of the ports, the MTU differences between ports, and whether those ports are on the same NUMA node. Just to complicate things a bit more, there are multiple overheads, and alignment and rounding need to be accounted for at various places in OVS-DPDK. Everything clear? OK, you can stop reading then!
However, if not, read on.

Continue reading “Open vSwitch-DPDK: How Much Hugepage Memory?”


What’s new in MACsec: setting up MACsec using wpa_supplicant and (optionally) NetworkManager

A few months ago, on this blog, we talked about MACsec. In this post, I want to introduce the work we’ve done since then. Since that work revolves around methods to configure MACsec, this will also act as a guide to configure it by two methods: wpa_supplicant alone, or NetworkManager with wpa_supplicant.

If you read the previous MACsec post, you probably thought that this whole business of generating keys and creating “secure associations” isn’t very convenient, especially given that you then have to monitor your associations and generate new keys manually. And you’re right: it’s not.

Besides, if you run RHEL or Fedora, you’re probably used to configuring your network with NetworkManager, so you would expect to be able to configure MACsec with NetworkManager as well. We’re going to describe this below. First, let’s go a little bit behind the scenes.

Continue reading “What’s new in MACsec: setting up MACsec using wpa_supplicant and (optionally) NetworkManager”


The need for speed and the kernel datapath – recent improvements in UDP packets processing

Networking hardware is becoming crazily fast, 10Gbs NICs are entry-level for server h/w, 100Gbs cards are increasingly popular and 200Gbs are already surfacing. While the Linux kernel is striving to cope with such speeds with large packets and all kind of aggregation, ISPs are requesting much more demanding workload with NFV and line rate packet processing even for 64 bytes packets.

Is everything lost and are we all doomed to rely on some kernel bypass solution? Possibly, but let’s first inspect what is really the current status for packet processing in the kernel data path, with a perspective look at the relevant history and the recent improvements.

We will focus on UDP packets reception: UDP flood is a common tool to stress the networking stack allowing arbitrary small packets and defeating packet aggregation (GRO), in place for other protocols.

Continue reading “The need for speed and the kernel datapath – recent improvements in UDP packets processing”


Open vSwitch: Overview of 802.1ad (QinQ) Support

Open vSwitch (OVS) recently gained support for 802.1ad (QinQ). It can be used as a lightweight alternative to tunnel technologies such as; VXLAN, GENEVE, GRE. A key advantage of QinQ is that it can make use of hardware offload features common in network interface cards (NICs). Only newer NICs support hardware offload for VXLAN and GENEVE. QinQ also incurs less frame processing and has a smaller encapsulation overhead.

Continue reading “Open vSwitch: Overview of 802.1ad (QinQ) Support”