Open vSwitch

Speeding up Open vSwitch with partial hardware offloading

Speeding up Open vSwitch with partial hardware offloading

Open vSwitch (OVS) can use the kernel datapath or the userspace datapath. There are interesting developments in the kernel datapath using hardware offloading through the TC Flower packet classifier, but in this article, the focus will be on the userspace datapath accelerated with the Data Plane Development Kit (DPDK) and its new feature—partial flow hardware offloading—to accelerate the virtual switch even more.

This article explains how the virtual switch worked before versus now and why the new feature can potentially save resources while improving the packet processing rate.

Continue reading “Speeding up Open vSwitch with partial hardware offloading”

Share
Performance improvements in OVN: Past and future

Performance improvements in OVN: Past and future

OVN (Open Virtual Network) is a subcomponent of Open vSwitch (OVS). It allows for the expression of overlay networks by connecting logical routers and logical switches. Cloud providers and cloud management systems have been using OVS for many years as a performant method for creating and managing overlay networks.

Lately, OVN has come into its own because it is being used more in Red Hat products. The result has been an increased amount of scrutiny for real-world scenarios with OVN. This has resulted in new features being added to OVN. More importantly, this has led to tremendous changes to improve performance in OVN.

In this article, I will discuss two game-changing performance improvements that have been added to OVN in the past year, and I will discuss future changes that we may see in the coming year.

Continue reading “Performance improvements in OVN: Past and future”

Share
IP packet buffering in OVN

IP packet buffering in OVN

Open Virtual Network (OVN) is a subproject of Open vSwitch (OVS), a performant, programmable, multi-platform virtual switch. OVN adds to the OVS existing capabilities the support for overlay networks by introducing virtual network abstractions such as virtual switches and routers. Moreover, OVN provides native methods for setting up Access Control Lists (ACLs) and network services such as DHCP. Many Red Hat products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN, and Red Hat OpenShift Container Platform will be using OVN soon.

In this article, I’ll cover how OVN ARP/ND_NS actions work, the main limitations in the current implementation, and how to overcome those. First, I’ll provide a brief overview of OVN’s architecture to facilitate the discussion:

Continue reading “IP packet buffering in OVN”

Share
Network debugging with eBPF (RHEL 8 Beta)

Network debugging with eBPF (RHEL 8 Beta)

Introduction

Networks are fun to work with, but often they are also a source of trouble. Network troubleshooting can be difficult, and reproducing the bad behavior that is happening in the field can be painful as well.

Luckily, there are some tools that come to the aid: network namespaces, virtual machines, tc, and netfilter. Simple network setups can be reproduced with network namespaces and veth devices, while more-complex setups require interconnecting virtual machines with a software bridge and using standard networking tools, like iptables or tc, to simulate the bad behavior. If you have an issue with ICMP replies generated because an SSH server is down, iptables -A INPUT -p tcp --dport 22 -j REJECT --reject-with icmp-host-unreachable in the correct namespace or VM can do the trick.

This article describes using eBPF (extended BPF), an extended version of the Berkeley Packet Filter, to troubleshoot complex network issues. eBPF is a fairly new technology and the project is still in an early stage, with documentation and the SDK not yet ready. But that should improve, especially with XDP (eXpress Data Path) being shipped in Red Hat Enterprise Linux 8 Beta, which you can download and run now.

Continue reading “Network debugging with eBPF (RHEL 8 Beta)”

Share
How to create an Open Virtual Network distributed gateway router

How to create an Open Virtual Network distributed gateway router

In this article, I discuss external connectivity in Open Virtual Network (OVN), a subproject of Open vSwitch (OVS), using a distributed gateway router.

OVN provides external connectivity in two ways:

  • A logical router with a distributed gateway port, which is referred to as a distributed gateway router in this article
  • A logical gateway router

In this article, you will see how to create a distributed gateway router and an example of how it works.

Creating a distributed gateway router has some advantages over using a logical gateway router for the CMS (cloud management system):

  • It is easier to create a distributed gateway router because the CMS doesn’t need to create a transit logical switch, which is needed for a logical gateway router.
  • A distributed gateway router supports distributed north/south traffic, whereas the logical gateway router is centralized on a single gateway chassis.
  • A distributed gateway router supports high availability.

Note: The CMS can be OpenStack, Red Hat OpenShift, Red Hat Virtualization, or any other system that manages a cloud.

Continue reading “How to create an Open Virtual Network distributed gateway router”

Share
Dynamic IP address management in Open Virtual Network (OVN): Part Two

Dynamic IP address management in Open Virtual Network (OVN): Part Two

In part one of this series, we explored the dynamic IP address management (IPAM) capabilities of Open Virtual Network. We covered the subnet, ipv6_prefix, and exclude_ips options on logical switches. We then saw how these options get applied to logical switch ports whose addresses have been set to the special “dynamic” value.  OVN, a subproject of Open vSwitch, is used for virtual networking in a number of Red Hat products like Red Hat OpenStack Platform, Red Hat Virtualization, and Red Hat OpenShift Container Platform in a future release.

In this part, we’re going to explore some of the oversights and downsides in the feature, how those have been corrected, and what’s in store for OVN in future versions.

Continue reading “Dynamic IP address management in Open Virtual Network (OVN): Part Two”

Share
Troubleshooting FDB table wrapping in Open vSwitch

Troubleshooting FDB table wrapping in Open vSwitch

When most people deploy an Open vSwitch configuration for virtual networking using the NORMAL rule, that is, using L2 learning, they do not think about configuring the size of the Forwarding DataBase (FDB).

When hardware-based switches are used, the FDB size is generally rather large and the large FDB size is a key selling point. However for Open vSwitch, the default FDB value is rather small, for example, in version 2.9 and earlier it is only 2K entries. Starting with version 2.10 the FDB size was increased to 8K entries. Note that for Open vSwitch, each bridge has its own FDB table for which the size is individually configurable.

This blog explains the effects of configuring too small an FDB table, how to identify which bridge is suffering from too small an FDB table, and how to configure the FDB table size appropriately.

Continue reading “Troubleshooting FDB table wrapping in Open vSwitch”

Share
Dynamic IP Address Management in Open Virtual Network (OVN): Part One

Dynamic IP Address Management in Open Virtual Network (OVN): Part One

Some background

For those unfamiliar, Open Virtual Network (OVN) is a subproject of OpenVswitch (OVS), a performant programmable multi-platform virtual switch. OVN provides the ability to express an overlay network as a series of virtual routers and switches. OVN also provides native methods for setting up Access Control Lists (ACLs), and it functions as an OpenFlow switch, providing services such as DHCP. The components of OVN program OVS on each of the hypervisors in the network. Many of Red Hat’s products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN. Red Hat OpenShift Container Platform will be using OVN soon.

Looking around the internet, it’s pretty easy to find high-quality tutorials on the basics of OVN. However, when it comes to more-advanced topics, it sometimes feels like the amount of information is lacking. In this tutorial, we’ll examine dynamic addressing in OVN. You will learn about IP address management (IPAM) options in OVN and how to apply them.

Continue reading “Dynamic IP Address Management in Open Virtual Network (OVN): Part One”

Share
Troubleshooting Open vSwitch DPDK PMD Thread Core Affinity

Troubleshooting Open vSwitch DPDK PMD Thread Core Affinity

The most common problem when people are trying to deploy an Open vSwitch with Data Plane Development Kit (OvS-DPDK) solution is that the performance is not as expected. For example, they are losing packets. This is where our journey for this series of blogs will start.

This first blog is about Poll Mode Driver (PMD) thread core affinity. It covers how to configure thread affinity and how to verify that it’s set up correctly. This includes making sure no other threads are using the CPU cores.

Continue reading “Troubleshooting Open vSwitch DPDK PMD Thread Core Affinity”

Share
Debugging Memory Issues with Open vSwitch DPDK

Debugging Memory Issues with Open vSwitch DPDK

Introduction

This article is about debugging out-of-memory issues with Open vSwitch with the Data Plane Development Kit (OvS-DPDK). It explains the situations in which you can run out of memory when using OvS-DPDK and it shows the log entries that are produced in those circumstances. It also shows some other log entries and commands for further debugging.

When you finish reading this article, you will be able to identify that you have an out-of-memory issue and you’ll know how to fix it. Spoiler: Usually having some more memory on the relevant NUMA node works. It is based on OvS 2.9.

Continue reading “Debugging Memory Issues with Open vSwitch DPDK”

Share