IP packet buffering in OVN

Open Virtual Network (OVN) is a subproject of Open vSwitch (OVS), a performant, programmable, multi-platform virtual switch. OVN adds to the OVS existing capabilities the support for overlay networks by introducing virtual network abstractions such as virtual switches and routers. Moreover, OVN provides native methods for setting up Access Control Lists (ACLs) and network services such as DHCP. Many Red Hat products, such as Red Hat OpenStack Platform and Red Hat Virtualization, are now using OVN, and Red Hat OpenShift Container Platform will be using OVN soon.

In this article, I’ll cover how OVN ARP/ND_NS actions work, the main limitations in the current implementation, and how to overcome those. First, I’ll provide a brief overview of OVN’s architecture to facilitate the discussion:

Continue reading “IP packet buffering in OVN”


Achieving high-performance, low-latency networking with XDP: Part I

XDP: From zero to 14 Mpps

In past years, the kernel community has been using different approaches in the quest for ever-increasing networking performance. While improvements have been measurable in several areas, a new wave of architecture-related security issues and related counter-measures has undone most of the gains, and purely in-kernel solutions for some packet-processing intensive workloads still lag behind the bypass solution, namely Data Plane Development Kit (DPDK), by almost an order of magnitude.

But the kernel community never sleeps (almost literally) and the holy grail of kernel-based networking performance has been found under the name of XDP: the eXpress Data Path. XDP is available in Red Hat Enterprise Linux 8 Beta, which you can download and run now.

Continue reading “Achieving high-performance, low-latency networking with XDP: Part I”


Network debugging with eBPF (RHEL 8 Beta)


Networks are fun to work with, but often they are also a source of trouble. Network troubleshooting can be difficult, and reproducing the bad behavior that is happening in the field can be painful as well.

Luckily, there are some tools that come to the aid: network namespaces, virtual machines, tc, and netfilter. Simple network setups can be reproduced with network namespaces and veth devices, while more-complex setups require interconnecting virtual machines with a software bridge and using standard networking tools, like iptables or tc, to simulate the bad behavior. If you have an issue with ICMP replies generated because an SSH server is down, iptables -A INPUT -p tcp --dport 22 -j REJECT --reject-with icmp-host-unreachable in the correct namespace or VM can do the trick.

This article describes using eBPF (extended BPF), an extended version of the Berkeley Packet Filter, to troubleshoot complex network issues. eBPF is a fairly new technology and the project is still in an early stage, with documentation and the SDK not yet ready. But that should improve, especially with XDP (eXpress Data Path) being shipped in Red Hat Enterprise Linux 8 Beta, which you can download and run now.

Continue reading “Network debugging with eBPF (RHEL 8 Beta)”


Managing containerized system services with Podman

In this article, I discuss containers, but look at them from another angle. We usually refer to containers as the best technology for developing new cloud-native applications and orchestrating them with something like Kubernetes. Looking back at the origins of containers, we’ve mostly forgotten that containers were born for simplifying application distribution on standalone systems.

In this article, we’ll talk about the use of containers as the perfect medium for installing applications and services on a Red Hat Enterprise Linux (RHEL) system. Using containers doesn’t have to be complicated, I’ll show how to run MariaDB, Apache HTTPD, and WordPress in containers, while managing those containers like any other service, through systemd and systemctl.

Additionally, we’ll explore Podman, which Red Hat has developed jointly with the Fedora community. If you don’t know what Podman is yet, see my previous article, Intro to Podman (Red Hat Enterprise Linux 7.6) and Tom Sweeney’s Containers without daemons: Podman and Buildah available in RHEL 7.6 and RHEL 8 Beta.

Continue reading “Managing containerized system services with Podman”


Solving the challenges of debugging microservices on a container platform

Microservices have become mainstream in the enterprise. This proliferation of microservices applications generates new problems, which requires a new approach to managing problems. A microservice is a small, independently deployable, and independently scalable software service that is designed to encapsulate a specific semantic function in the larger applicationl. This article explores several approaches to deploying tools to debug microservices applications on a Kubernetes platform like Red Hat OpenShift, including OpenTracing,  Squash, Telepresence, and creating a Squash Operator in Red Hat Ansible Automation.

Continue reading “Solving the challenges of debugging microservices on a container platform”


Extending support to Spring Boot for Red Hat OpenShift Application Runtimes

What Red Hat is providing

Red Hat OpenShift Application Runtimes (RHOAR) is a recommended set of products, tools, and components for developing and maintaining cloud-native applications on the Red Hat OpenShift platform. As part of this offering, Red Hat is extending its support to Spring Boot and related frameworks for building modern, production-grade, Java-based cloud-native applications.

Spring Boot lets you create opinionated Spring-based standalone applications. The Spring Boot runtime also integrates with the OpenShift platform, allowing your services to externalize their configuration, implement health checks, provide resiliency and failover, and much more. To learn more about how Spring Boot applications integrate with the wider Red Hat portfolio, check out the following OpenShift Commons Briefing by Thomas Qvarnstrom:

Continue reading “Extending support to Spring Boot for Red Hat OpenShift Application Runtimes”


The Non-complexity of /etc/nsswitch.conf

In most glibc-based operating systems, there’s a file /etc/nsswitch.conf that most people ignore, few people understand, but all people generally rely on. This file determines where the system finds things like host names, passwords, and protocol numbers. Does your company use LDAP? NIS? Plain files? The nsswitch file (it stands for “name services switch”) tells the system what service to use for each type of name lookup.

Here’s a snippet from a sample /etc/nsswitch.conf file:

passwd:   files nis
group:    files nis

hosts:    files dns myhostname

In this example, user information (the passwd and group services) come first from “files” (like /etc/passwd or /etc/group), and if no entries are found there, a query to an NIS server (configured elsewhere) will be used. Host information first comes from /etc/hosts (files), then a DNS server (dns), and if neither of those work, at least a fallback of “myhostname” so that the local machine has some name.

The non-complexity comes in the “and if that doesn’t work” rule. When multiple services are listed, they’re tried in order, and a sevice either succeeds or fails. If it fails, the next is tried, etc. There’s a way to check for a few other cases (entry not found, or service temporarily unavailable) but the only thing you can do based on those is either try the next service, or don’t.

This simplicity has already led to one drawback, which is the handling of a user’s groups when there are both local groups and global (LDAP for example) groups. You wouldn’t want to have to choose between the two, you’d rather be allowed into the union of the two sets of groups. The nsswitch.conf syntax has a special case for that:

groups:   files [SUCCESS=merge] ldap

Normally, a successful lookup would terminate the lookup and return a value. This special case says that even if the “files” lookup worked, try the “ldap” service also and merge the two lists of groups together. Another special case has come up recently, with systemd providing some services for the local machine. The problem here is “local machine” – it doesn’t provide global services, but the nsswitch syntax doesn’t allow for “delegation”. If the systemd service doesn’t find a record, is that “not found” authoritative?

And what if you wanted different services for different domains? Currently, your only option is to configure a local DNS server that has all the logic for domain delegation, and refer to that in nsswitch.conf. That only works if the DNS server is running, which may not be the case when the machine is first booting, and may not be easy to configure when you change networks (wifi? vpn?).

So what’s the point of this article? I’ve been contemplating the non-complexity of this file, and wondering what its future might contain. What if you could select services based on a pattern match of the search string? What if it could test for outside conditions like time of day, or VPNs? What if your corporate security rules required a more deterministic failure mode? I’m not suggesting nsswitch.conf become some full programming language (nsswitch.py anyone? 😉 but if it had some decision-making syntax beyond what it already has, what would it look like? What kinds of problems would it solve? Do you have any problems that could be solved by a more flexible nsswitch.conf? If so, share them in the comments below!


Leveraging OpenShift or Kubernetes for automated performance tests (part 1)

This is the first article in a series of three articles based on a session I hold at Red Hat Tech Exchange EMEA. In this first article, I present the rationale and approach for leveraging Red Hat OpenShift or Kubernetes for automated performance testing, give an overview of the setup, and discuss points that are worth considering when executing and analyzing performance tests. I will also say a few words about performance tuning.

In the second article, we will look at building an observability stack, which—beyond the support it provides in production—can be leveraged during performance tests. Open sources projects like Prometheus, Jaeger, Elasticsearch and Grafana will be used for the purpose. The third article will present the details for building an environment for performance testing and automating the execution with JMeter and Jenkins.

Continue reading “Leveraging OpenShift or Kubernetes for automated performance tests (part 1)”


Red Hat Enterprise Linux 8 Beta is here

Red Hat Enterprise Linux 8 Beta is here

And it’s been built with production stability and development agility in mind.

There’s so much to say about RHEL 8 Beta, but I want to focus on just a few points from the corporate announcement that highlight Red Hat Enterprise Linux 8 Beta as a developer platform that:

  • Simplifies application development – with less setup and config effort, you can more quickly get to writing code
  • Is the easiest RHEL yet for developers that are new to Linux
  • Is for traditional and cloud/container applications with many new tools for both
  • Already delivers dozens of tools to build and test applications

Now let’s zoom in on what these mean.

Download the RHEL 8 Beta here. 

“Too slow, too fast”

We’ve gotten this feedback a lot when discussing availability and support of development packages. To address this dichotomy, Red Hat Enterprise Linux 8 Beta has built in the concept of Application Streams to deliver userspace packages (programming languages, compilers, databases, etc.) more simply and with greater flexibility – this addresses the “too slow”.  For the “too fast” requirement, there are also “Core” components that have the same lifecycle as the operating system – 10 years. Users will often have a version in each grouping. Application Streams – think of them as “son of Software Collections” – are a simpler way to deliver this modularity with improved installation, use, and re-use

Continue reading “Red Hat Enterprise Linux 8 Beta is here”


Introducing CodeReady Linux Builder

The RHEL8 Beta introduces a new repository, the CodeReady Linux Builder (or “Builder” for short) that developers may need while developing applications for RHEL. As you all know “developer” is not a one size fits all term. As a result, I am taking this opportunity to try to explain when you might need Builder enabled for your development activities.

First off, if you are a typical web developer, dealing with PHP, Ruby, or Perl you are unlikely to need the content delivered through Builder. The PHP packages, Ruby gems, and Perl modules provided in the AppStream repository will, in most cases, provide sufficient functionality to develop and run applications you create yourself and to run frameworks like Drupal, WordPress, Rails, or Twiki.  Please see the appropriate HowTos for getting these things up and running.

Ruby and Perl both have additional libraries made available in the Builder repository. However, they are less commonly used or used at build time only.

Next we have Java developers. Again, much of the functionality and jars you would expect to use normally have been provided in the AppStream. For example ant, maven and apache-commons-logging can be found directly in AppStream. However, if you need some of the build-only components, you would find those in the Builder repository.

If you are a .Net developer, you can find the Core Runtime & tools directly in AppStream as the “dotnet” package. When you build applications you will be pulling most of your dependencies from Microsoft or the upstream of those dependencies. As a .Net developer, you will not need the Builder repository.

Moving on to the traditionally compiled languages, the Builder repository is really targeted to you. For languages like C and C++, many of the header files, devel packages, etc. can be found in the Builder repository.  As this sort of developer, you will definitely want to have the Builder repository enabled on your build machines. However, you should not, normally, need the repository enabled on your runtime deployments.

Much like .Net, the LLVM/Clang, Go & Rust language compilers are provided directly in AppStream with a few tools to support development. If you use one of these languages, you won’t need the Builder repository.

Last but not least, when you want to package and deploy your applications, you can find many of the tools that support you in this process in the Builder repository as well. For example, meson, dejagnu, and doxygen are available for use.

Hopefully, you found this description of the new Code Ready Linux Builder helpful and we really hope the changes to the content repositories with RHEL8 make things simpler and easier to find.