networking

IP packet buffering in OVN

IP packet buffering in OVN

Open Virtual Network (OVN) is a subproject of Open vSwitch (OVS), a performant, programmable, multi-platform virtual switch. OVN adds to the OVS existing capabilities the support for overlay networks by introducing virtual network abstractions such as virtual switches and routers. Moreover, OVN provides native methods for setting up Access Control Lists (ACLs) and network services such as DHCP. Many Red Hat products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN, and Red Hat OpenShift Container Platform will be using OVN soon.

In this article, I’ll cover how OVN ARP/ND_NS actions work, the main limitations in the current implementation, and how to overcome those. First, I’ll provide a brief overview of OVN’s architecture to facilitate the discussion:

Continue reading “IP packet buffering in OVN”

Share
Achieving high-performance, low-latency networking with XDP: Part I

Achieving high-performance, low-latency networking with XDP: Part I

XDP: From zero to 14 Mpps

In past years, the kernel community has been using different approaches in the quest for ever-increasing networking performance. While improvements have been measurable in several areas, a new wave of architecture-related security issues and related counter-measures has undone most of the gains, and purely in-kernel solutions for some packet-processing intensive workloads still lag behind the bypass solution, namely Data Plane Development Kit (DPDK), by almost an order of magnitude.

But the kernel community never sleeps (almost literally) and the holy grail of kernel-based networking performance has been found under the name of XDP: the eXpress Data Path. XDP is available in Red Hat Enterprise Linux 8 Beta, which you can download and run now.

Continue reading “Achieving high-performance, low-latency networking with XDP: Part I”

Share
Network debugging with eBPF (RHEL 8 Beta)

Network debugging with eBPF (RHEL 8 Beta)

Introduction

Networks are fun to work with, but often they are also a source of trouble. Network troubleshooting can be difficult, and reproducing the bad behavior that is happening in the field can be painful as well.

Luckily, there are some tools that come to the aid: network namespaces, virtual machines, tc, and netfilter. Simple network setups can be reproduced with network namespaces and veth devices, while more-complex setups require interconnecting virtual machines with a software bridge and using standard networking tools, like iptables or tc, to simulate the bad behavior. If you have an issue with ICMP replies generated because an SSH server is down, iptables -A INPUT -p tcp --dport 22 -j REJECT --reject-with icmp-host-unreachable in the correct namespace or VM can do the trick.

This article describes using eBPF (extended BPF), an extended version of the Berkeley Packet Filter, to troubleshoot complex network issues. eBPF is a fairly new technology and the project is still in an early stage, with documentation and the SDK not yet ready. But that should improve, especially with XDP (eXpress Data Path) being shipped in Red Hat Enterprise Linux 8 Beta, which you can download and run now.

Continue reading “Network debugging with eBPF (RHEL 8 Beta)”

Share
How to create an Open Virtual Network distributed gateway router

How to create an Open Virtual Network distributed gateway router

In this article, I discuss external connectivity in Open Virtual Network (OVN), a subproject of Open vSwitch (OVS), using a distributed gateway router.

OVN provides external connectivity in two ways:

  • A logical router with a distributed gateway port, which is referred to as a distributed gateway router in this article
  • A logical gateway router

In this article, you will see how to create a distributed gateway router and an example of how it works.

Creating a distributed gateway router has some advantages over using a logical gateway router for the CMS (cloud management system):

  • It is easier to create a distributed gateway router because the CMS doesn’t need to create a transit logical switch, which is needed for a logical gateway router.
  • A distributed gateway router supports distributed north/south traffic, whereas the logical gateway router is centralized on a single gateway chassis.
  • A distributed gateway router supports high availability.

Note: The CMS can be OpenStack, Red Hat OpenShift, Red Hat Virtualization, or any other system that manages a cloud.

Continue reading “How to create an Open Virtual Network distributed gateway router”

Share
Dynamic IP address management in Open Virtual Network (OVN): Part Two

Dynamic IP address management in Open Virtual Network (OVN): Part Two

In part one of this series, we explored the dynamic IP address management (IPAM) capabilities of Open Virtual Network. We covered the subnet, ipv6_prefix, and exclude_ips options on logical switches. We then saw how these options get applied to logical switch ports whose addresses have been set to the special “dynamic” value.  OVN, a subproject of Open vSwitch, is used for virtual networking in a number of Red Hat products like Red Hat OpenStack Platform, Red Hat Virtualization, and Red Hat OpenShift Container Platform in a future release.

In this part, we’re going to explore some of the oversights and downsides in the feature, how those have been corrected, and what’s in store for OVN in future versions.

Continue reading “Dynamic IP address management in Open Virtual Network (OVN): Part Two”

Share
Troubleshooting FDB table wrapping in Open vSwitch

Troubleshooting FDB table wrapping in Open vSwitch

When most people deploy an Open vSwitch configuration for virtual networking using the NORMAL rule, that is, using L2 learning, they do not think about configuring the size of the Forwarding DataBase (FDB).

When hardware-based switches are used, the FDB size is generally rather large and the large FDB size is a key selling point. However for Open vSwitch, the default FDB value is rather small, for example, in version 2.9 and earlier it is only 2K entries. Starting with version 2.10 the FDB size was increased to 8K entries. Note that for Open vSwitch, each bridge has its own FDB table for which the size is individually configurable.

This blog explains the effects of configuring too small an FDB table, how to identify which bridge is suffering from too small an FDB table, and how to configure the FDB table size appropriately.

Continue reading “Troubleshooting FDB table wrapping in Open vSwitch”

Share
Dynamic IP Address Management in Open Virtual Network (OVN): Part One

Dynamic IP Address Management in Open Virtual Network (OVN): Part One

Some background

For those unfamiliar, Open Virtual Network (OVN) is a subproject of OpenVswitch (OVS), a performant programmable multi-platform virtual switch. OVN provides the ability to express an overlay network as a series of virtual routers and switches. OVN also provides native methods for setting up Access Control Lists (ACLs), and it functions as an OpenFlow switch, providing services such as DHCP. The components of OVN program OVS on each of the hypervisors in the network. Many of Red Hat’s products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN. Red Hat OpenShift Container Platform will be using OVN soon.

Looking around the internet, it’s pretty easy to find high-quality tutorials on the basics of OVN. However, when it comes to more-advanced topics, it sometimes feels like the amount of information is lacking. In this tutorial, we’ll examine dynamic addressing in OVN. You will learn about IP address management (IPAM) options in OVN and how to apply them.

Continue reading “Dynamic IP Address Management in Open Virtual Network (OVN): Part One”

Share
Debugging Memory Issues with Open vSwitch DPDK

Debugging Memory Issues with Open vSwitch DPDK

Introduction

This article is about debugging out-of-memory issues with Open vSwitch with the Data Plane Development Kit (OvS-DPDK). It explains the situations in which you can run out of memory when using OvS-DPDK and it shows the log entries that are produced in those circumstances. It also shows some other log entries and commands for further debugging.

When you finish reading this article, you will be able to identify that you have an out-of-memory issue and you’ll know how to fix it. Spoiler: Usually having some more memory on the relevant NUMA node works. It is based on OvS 2.9.

Continue reading “Debugging Memory Issues with Open vSwitch DPDK”

Share
Non-root Open vSwitch in RHEL

Non-root Open vSwitch in RHEL

In a few weeks, the Fast Datapath Production channel will update the Open vSwitch version from the 2.7 series to the 2.9 series. This is an important change in more ways than one. A wealth of new features and fixes all related to packet movement will come into play. One that will surely be blamed for all your troubles will be the integration of the `–ovs-user` flag to allow for an unprivileged user to interact with Open vSwitch.

Running as root can solve a lot of pesky problems. Want to write to an arbitrary file? No problem. Want to load kernel modules? Go for it! Want to sniff packets on the wire? Have a packet dump. All of these are great when the person commanding the computer is the rightful owner. But the moment the person in front of the keyboard isn’t the rightful owner, problems occur.

Continue reading “Non-root Open vSwitch in RHEL”

Share
Open vSwitch-DPDK: How Much Hugepage Memory?

Open vSwitch-DPDK: How Much Hugepage Memory?

Introduction

In order to maximize performance of the Open vSwitch DPDK datapath, it pre-allocates hugepage memory. As a user you are responsible for telling Open vSwitch how much hugepage memory to pre-allocate. The question of exactly what value to use often arises. The answer is, it depends.

There is no simple answer as it depends on things like the MTU size of the ports, the MTU differences between ports, and whether those ports are on the same NUMA node. Just to complicate things a bit more, there are multiple overheads, and alignment and rounding need to be accounted for at various places in OVS-DPDK. Everything clear? OK, you can stop reading then!
However, if not, read on.

Continue reading “Open vSwitch-DPDK: How Much Hugepage Memory?”

Share