It isn’t always easy to understand how Open vSwitch (OVS) cycles are spent, especially because various parameters and configuration options can affect how OVS behaves. Members of the Open vSwitch community are actively working to understand what causes packets drops in Open vSwitch. Efforts so far have included adding a custom statistic for vHost TX retries, tracking vHost TX contention, and adding a coverage counter to count vHost IRQs. We are particularly interested in the user space datapath that uses the Data Plane Development Kit (DPDK) for fast I/O.
Adding these statistics is an ongoing effort, and we won’t cover all of the corners. In some cases, the statistics leave doubts about what is causing a behavior.
In this article, I will introduce a new counter we’ve added to learn more about contention in the vHost transmission path. I’ll also show you how to use the new counter with
perf, and I’ll discuss what’s next for our ongoing efforts.
Continue reading “Debugging vHost user TX contention in Open vSwitch”
After loads of email and IRC discussions, the Open Virtual Network (OVN) source code has been separated from the Open vSwitch (OVS) source code, and the two projects now operate independently. In this article, we’ll explain the reasons for separating OVN from OVS, the technical aspects of the split, and the upcoming challenges for the OVN project.
Continue reading “The clean break of Open Virtual Network from Open vSwitch”
Open vSwitch (OVS) can use the kernel datapath or the userspace datapath. There are interesting developments in the kernel datapath using hardware offloading through the TC Flower packet classifier, but in this article, the focus will be on the userspace datapath accelerated with the Data Plane Development Kit (DPDK) and its new feature—partial flow hardware offloading—to accelerate the virtual switch even more.
This article explains how the virtual switch worked before versus now and why the new feature can potentially save resources while improving the packet processing rate.
Continue reading “Speeding up Open vSwitch with partial hardware offloading”
OVN (Open Virtual Network) is a subcomponent of Open vSwitch (OVS). It allows for the expression of overlay networks by connecting logical routers and logical switches. Cloud providers and cloud management systems have been using OVS for many years as a performant method for creating and managing overlay networks.
Lately, OVN has come into its own because it is being used more in Red Hat products. The result has been an increased amount of scrutiny for real-world scenarios with OVN. This has resulted in new features being added to OVN. More importantly, this has led to tremendous changes to improve performance in OVN.
In this article, I will discuss two game-changing performance improvements that have been added to OVN in the past year, and I will discuss future changes that we may see in the coming year.
Continue reading “Performance improvements in OVN: Past and future”
Open Virtual Network (OVN) is a subproject of Open vSwitch (OVS), a performant, programmable, multi-platform virtual switch. OVN adds to the OVS existing capabilities the support for overlay networks by introducing virtual network abstractions such as virtual switches and routers. Moreover, OVN provides native methods for setting up Access Control Lists (ACLs) and network services such as DHCP. Many Red Hat products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN, and Red Hat OpenShift Container Platform will be using OVN soon.
In this article, I’ll cover how OVN ARP/ND_NS actions work, the main limitations in the current implementation, and how to overcome those. First, I’ll provide a brief overview of OVN’s architecture to facilitate the discussion:
Continue reading “IP packet buffering in OVN”
Networks are fun to work with, but often they are also a source of trouble. Network troubleshooting can be difficult, and reproducing the bad behavior that is happening in the field can be painful as well.
Luckily, there are some tools that come to the aid: network namespaces, virtual machines,
netfilter. Simple network setups can be reproduced with network namespaces and
veth devices, while more-complex setups require interconnecting virtual machines with a software bridge and using standard networking tools, like
tc, to simulate the bad behavior. If you have an issue with ICMP replies generated because an SSH server is down,
iptables -A INPUT -p tcp --dport 22 -j REJECT --reject-with icmp-host-unreachable in the correct namespace or VM can do the trick.
This article describes using eBPF (extended BPF), an extended version of the Berkeley Packet Filter, to troubleshoot complex network issues. eBPF is a fairly new technology and the project is still in an early stage, with documentation and the SDK not yet ready. But that should improve, especially with XDP (eXpress Data Path) being shipped in Red Hat Enterprise Linux 8, which you can download and run now.
Continue reading “Network debugging with eBPF (RHEL 8)”
In this article, I discuss external connectivity in Open Virtual Network (OVN), a subproject of Open vSwitch (OVS), using a distributed gateway router.
OVN provides external connectivity in two ways:
- A logical router with a distributed gateway port, which is referred to as a distributed gateway router in this article
- A logical gateway router
In this article, you will see how to create a distributed gateway router and an example of how it works.
Creating a distributed gateway router has some advantages over using a logical gateway router for the CMS (cloud management system):
- It is easier to create a distributed gateway router because the CMS doesn’t need to create a transit logical switch, which is needed for a logical gateway router.
- A distributed gateway router supports distributed north/south traffic, whereas the logical gateway router is centralized on a single gateway chassis.
- A distributed gateway router supports high availability.
Note: The CMS can be OpenStack, Red Hat OpenShift, Red Hat Virtualization, or any other system that manages a cloud.
Continue reading “How to create an Open Virtual Network distributed gateway router”
In part one of this series, we explored the dynamic IP address management (IPAM) capabilities of Open Virtual Network. We covered the
exclude_ips options on logical switches. We then saw how these options get applied to logical switch ports whose addresses have been set to the special “dynamic” value. OVN, a subproject of Open vSwitch, is used for virtual networking in a number of Red Hat products like Red Hat OpenStack Platform, Red Hat Virtualization, and Red Hat OpenShift Container Platform in a future release.
In this part, we’re going to explore some of the oversights and downsides in the feature, how those have been corrected, and what’s in store for OVN in future versions.
Continue reading “Dynamic IP address management in Open Virtual Network (OVN): Part Two”
When most people deploy an Open vSwitch configuration for virtual networking using the NORMAL rule, that is, using L2 learning, they do not think about configuring the size of the Forwarding DataBase (FDB).
When hardware-based switches are used, the FDB size is generally rather large and the large FDB size is a key selling point. However for Open vSwitch, the default FDB value is rather small, for example, in version 2.9 and earlier it is only 2K entries. Starting with version 2.10 the FDB size was increased to 8K entries. Note that for Open vSwitch, each bridge has its own FDB table for which the size is individually configurable.
This blog explains the effects of configuring too small an FDB table, how to identify which bridge is suffering from too small an FDB table, and how to configure the FDB table size appropriately.
Continue reading “Troubleshooting FDB table wrapping in Open vSwitch”
For those unfamiliar, Open Virtual Network (OVN) is a subproject of OpenVswitch (OVS), a performant programmable multi-platform virtual switch. OVN provides the ability to express an overlay network as a series of virtual routers and switches. OVN also provides native methods for setting up Access Control Lists (ACLs), and it functions as an OpenFlow switch, providing services such as DHCP. The components of OVN program OVS on each of the hypervisors in the network. Many of Red Hat’s products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN. Red Hat OpenShift Container Platform will be using OVN soon.
Looking around the internet, it’s pretty easy to find high-quality tutorials on the basics of OVN. However, when it comes to more-advanced topics, it sometimes feels like the amount of information is lacking. In this tutorial, we’ll examine dynamic addressing in OVN. You will learn about IP address management (IPAM) options in OVN and how to apply them.
Continue reading “Dynamic IP Address Management in Open Virtual Network (OVN): Part One”