Article

Customizing and tuning the Kuryr SDN for Red Hat OpenShift 3.11 on Red Hat OpenStack 13

October 2, 2020
Containers Kubernetes Linux
Mohammad Ahmad
Consultant
Luis Tomas Bolivar
Principal Software Engineer
Featured image for Kuryr SDN, OpenShift, and OpenStack

In a previous article, I showed you how to customize Red Hat OpenShift software-defined networking (SDN) for your organization's requirements and restrictions. In this article, we'll look at using the Kuryr SDN instead. Using Kuryr with OpenShift 3.11 on Red Hat OpenStack 13 changes the customization requirements because Kuryr works directly with OpenStack Neutron and Octavia.

Note: This article builds on the discussion and examples from my previous one. I recommend reading the previous one first.

Background

Traditional OpenShift installations leverage openshift-sdn, which is specific to OpenShift. Using openshift-sdn means that your containers run on a network within a network. This setup, known as double encapsulation, introduces an additional layer of complexity, which becomes apparent when troubleshooting network issues. Double encapsulation also affects network performance due to the overhead of running a network within a network.

If you are running OpenShift on OpenStack, then you have the option to use the Kuryr SDN, which allows you to directly access OpenStack's Neutron services and avoid double encapsulation. Using the kuryr-cni means that all of your OpenShift components—networks, subnets, load balancers, ports, and so on—are, in fact, OpenStack resources. This setup reduces the complexity of the networking layer. It also improves network performance. As another benefit, you can assign floating IPs to any of the OpenShift services, which allows traffic directly into the service without having to go through the OpenShift router.

In some cases, you might need to customize Kuryr's network-range defaults, such as if the defaults overlap with your organization’s network. Let's look at the requirements for customizing the Kuryr network.

Customizing the Kuryr network

We'll use the same base address range from the customization example in my previous article: 192.168.0.0/16. In this case, we'll divide the range between the service network (192.168.128.0/18), pod network (192.168.0.0/17), and docker-bridge network. For this specific example, we intend to configure Kuryr with namespace isolation, similar to ovs-multitenant when using openshift-sdn.

Note: OpenShift-Ansible, which was featured in my last article, has significantly more parameter requirements to deliver the same network IP range customizations that we can achieve with the Kuryr Container Network Interface (CNI).

Kuryr service network

Because Kuryr uses Octavia load balancers (which is an Amphora VM in OpenStack), every OpenShift service is an instance of an Octavia load balancer. Therefore, openshift_portal_net is the address range that OpenShift uses to assign IP addresses to OpenShift (Kubernetes) services. Each IP address will be associated with an Octavia load balancer as a VIP (Virtual IP, or floating port in OpenStack) in case of failover. Additionally, because Octavia load balancers are Amphora VMs, they also require a port/IP, which should not collide with the openshift_portal_net range.

Using the following ranges ensures a clear distinction between the Amphora VM IP range and the OpenShift service IP range:

  • openshift_openstack_kuryr_service_subnet_cidr: The range to be used by both OpenShift service IPs and the Octavia load balancer Amphora VM IPs. In this case, the range is 192.168.128.0/18, which is between 192.168.128.0 and 192.168.191.254.
  • openshift_portal_net: The range dedicated to OpenShift service IPs. In this case, the range is 192.168.128.0/19, which is between 192.168.128.0 and 192.168.159.254.
  • openshift_openstack_kuryr_service_pool_start: The start of the range to be used by the Octavia load balancer Amphora VM IPs. In this case, it is the second half of the range identified in the first parameter (openshift_openstack_kuryr_service_subnet_cidr). The start of that range is 192.168.160.0.
  • openshift_openstack_kuryr_service_pool_end: The end of the range used by the Octavia load balancer Amphora VM IPs. In this case, it is 192.168.191.254.

Figure 1 shows Amphora VM's IP allocation when implemented with an OpenShift service.

Figure 1: Clearly defining each range avoids IP collisions.

Kuryr pod network

There are no changes to the pod network, which is straightforward. In our case, it's: 

openshift_openstack_kuryr_pod_subnet_cidr: 192.168.0.0/17

Customizing the Kuryr inventory parameters

Next, we'll look at all of the inventory parameters required to successfully configure a customized OpenShift IP range.

Enabling namespace isolation

These parameters are needed to enable namespace isolation in Kuryr: 

openshift_use_kuryr: True
openshift_use_openshift_sdn: False
use_trunk_ports: True
os_sdn_network_plugin_name: cni
openshift_node_proxy_mode: userspace
kuryr_openstack_pool_driver: nested

You must ensure that os_sdn_network_plugin_name is unset.

IP range

Failing to set the IP range clearly causes performance problems with the kuryr-controller, and IP conflicts: 

#SERVICE:

openshift_portal_net: 192.168.128.0/19
openshift_openstack_kuryr_service_subnet_cidr: 192.168.128.0/18
openshift_openstack_kuryr_service_pool_start: 192.168.160.1
openshift_openstack_kuryr_service_pool_end: 192.168.191.253

#POD
openshift_openstack_kuryr_pod_subnet_cidr: 192.168.0.0/17

The OpenShift service IP range is from 192.168.128.0 to 192.168.159.254; the Amphora VM IP range is from 192.168.160.1 to 19.168.191.254; and the pod IP range is from 192.168.0.1 to 192.168.127.254.

Note that the following types of errors will cause the OpenShift service IP range to conflict with Amphora VM's IP range: 

ERROR kuryr_kubernetes.controller.drivers.lbaasv2 [-]
Error when creating loadbalancer: {"debuginfo": null, "faultcode":
 "Client", "faultstring": "IP address 192.168.123.123 already
allocated in subnet

Pre-creating subports

This value is generally relevant when you have a flat network, where pre-creating subports contributes to the overall deployment speed for applications that require network resources. In most cases, you can set this value to false: 

openshift_kuryr_precreate_subports: false

Cluster sizing

This parameter changes the way that you size a cluster, so it is important. The number of pods per namespace is determined by prefixlen: 

openshift_openstack_kuryr_pod_subnet_prefixlen

Where:

  • /24 = 256 pods per namespace
  • /25 = 128 pods per namespace
  • /26 = 64 pods per namespace

Pool batch

The kuryr_openstack_pool_batch value needs to be set based on openshift_openstack_kuryr_pod_subnet_prefixlen: 

kuryr_openstack_pool_max:

kuryr_openstack_pool_min:

Here are some suggestions for the following values of openshift_openstack_kuryr_pod_subnet_prefixlen:

24: batch: 5, max: 10, min: 1
25: batch: 4, max: 7, min: 1
26: batch: 3, max: 5, min: 1

Also note that each OpenShift node will have its own pool, so if you have three worker nodes, each node will have a dedicated pool for that namespace. This requirement limits how many worker nodes you can use based on the size of openshift_openstack_kuryr_pod_subnet_prefixlen.

kuryr_openstack_ca: "MYORG_CA_Bundle.txt"

If you do not set this value, Kuryr will fail.

Images

Providing specific values for the images rules out the risk of default images being incorrect: 

openshift_openstack_kuryr_controller_image

openshift_openstack_kuryr_cni_image

kuryr_openstack_public_net_id

You can get this value from openstack network list.

Global namespaces

By default, these are the only namespaces that are considered global, meaning that Kuryr allows other namespaces to reach these namespaces despite namespace isolation: 

kuryr_openstack_global_namespaces: default,openshift-monitoring

Namespace isolation

The openshift_kuryr_subnet_driver: namespace and openshift_kuryr_sg_driver: namespace are required for namespace isolation.

DNS lookup

The values here allow DNS lookup: 

openshift_master_open_ports:
- service: dns tcp
  port: 53/tcp
- service: dns udp
  port: 53/udp
openshift_node_open_ports:
- service: dns tcp
  port: 53/tcp
- service: dns udp
  port: 53/udp

openshift_openstack_node_secgroup_rules

# NOTE(shadower): the 53 rules are needed for Kuryr
  - direction: ingress
    protocol: tcp
    port_range_min: 53
    port_range_max: 53
  - direction: ingress
    protocol: udp
    port_range_min: 53
    port_range_max: 53
  - direction: ingress
    protocol: tcp
    port_range_min: 10250
    port_range_max: 10250
    remote_ip_prefix: "{{ openshift_openstack_kuryr_pod_subnet_cidr }}"
  - direction: ingress
    protocol: tcp
    port_range_min: 10250
    port_range_max: 10250
    remote_ip_prefix: "{{ openshift_openstack_subnet_cidr }}"
  - direction: ingress
    protocol: udp
    port_range_min: 4789
    port_range_max: 4789
    remote_ip_prefix: "{{ openshift_openstack_kuryr_pod_subnet_cidr }}"
  - direction: ingress
    protocol: udp
    port_range_min: 4789
    port_range_max: 4789
    remote_ip_prefix: "{{ openshift_openstack_subnet_cidr }}"
  - direction: ingress
    protocol: tcp
    port_range_min: 9100
    port_range_max: 9100
    remote_ip_prefix: "{{ openshift_openstack_kuryr_pod_subnet_cidr }}"
  - direction: ingress
    protocol: tcp
    port_range_min: 9100
    port_range_max: 9100
    remote_ip_prefix: "{{ openshift_openstack_subnet_cidr }}"
  - direction: ingress
    protocol: tcp
    port_range_min: 8444
    port_range_max: 8444
    remote_ip_prefix: "{{ openshift_openstack_kuryr_pod_subnet_cidr }}"
  - direction: ingress
    protocol: tcp
    port_range_min: 8444
    port_range_max: 8444
    remote_ip_prefix: "{{ openshift_openstack_subnet_cidr }}"

Most importantly, never use remote_group_id. Instead, only use remote_ip_prefix.

Conclusion

Using Kuryr with OpenShift 3.11 on OpenStack 13 provides the benefits of using OpenStack's Neutron networking directly for OpenShift pods and services, instead of the OpenShift SDN. Avoiding double encapsulation improves performance and reduces troubleshooting complexity, and you also get the benefits of directly associating floating IPs to OpenShift services, which is useful in several applications.

There are, however, parameters that must be configured correctly in order to get an optimally running OpenShift cluster with Kuryr. This article identified all of the important parameters and the recommended values that are required to configure a cluster that suits your needs.

Acknowledgments

I would like to acknowledge Luis Tomas Bolivar as my co-author and Phuong Nguyen as our peer reviewer.

References

For additional information:

Comments