In this article, I'll highlight a practical case for customizing the Red Hat OpenShift software-defined network (SDN). To achieve this, I will identify the OpenShift-Ansible inventory parameters that configure different aspects of the OpenShift SDN, specifically the cluster, portal, and docker networks.
Why customize the SDN?
An important question I am often asked is: Why do you need to customize the SDN? Isn’t it completely internal? Users generally assume there is no need to customize the SDN because OpenShift’s SDN has no impact on networks outside the OpenShift cluster; therefore, IP conflicts should not be a concern. However, this is not always the case.
In large organizations, designated private networks (e.g., 10.0.0.0/8) are often used “publicly” within the internal confines of the organization's own network. With that in mind, the primary concern is that services external to the OpenShift cluster may be using IPs within the normally private ranges that conflict with those used by the cluster itself.
These conflicts will cause issues when applications in the OpenShift cluster attempt to communicate to these conflicting external services, as the OpenShift SDN will attempt to route these requests within its own network, causing these services to be unreachable from within the cluster.
Background
Default values
The default networks configured by OpenShift at installation time are:
| Component | Ansible inventory parameters | Notes |
| The cluster network | osm_cluster_network_cidr and osm_host_subnet_length which by default are 10.128.0.0/14 and 9 respectively.
|
Each node gets a “/23” subnet from this range, allowing a total of 512 subnets dedicated for nodes, and a total of 510 IPs on each node for containers. Running “oc get pods --all-namespaces -o wide” will show which IPs have been allocated from this range, and should all be within this range. |
| The services network | openshift_portal_net | By default this is 172.30.0.0/16. This means you have a total of 65534 addresses to assign to services.
Running “oc get services --all-namespaces” should also show that all service IPs are within this range. |
| The docker network | openshift_docker_options="--bip 172.17.0.1/16 --fixed-cidr 172.17.0.0/17" | These values are command line arguments to the docker daemon configured by the openshift_docker_options variable in openshift-ansible. |
Customizing these values
Fine-tuning each of these values dictates how large your cluster can grow by way of defining how many nodes, pods, and services it can handle. With that in mind, for a development cluster, you may want to allow more services, as opposed to pods, to address specific needs required by developers. However, on a production system, you may prefer the ability to scale in the number of nodes to address a business and operational requirement.
This article is based on the situation where you only have the subnet 192.168.0.0/16 for the entire cluster to use. With that in mind, I partition the subnet as follows:
| Subnet | Use |
| 192.168.0.0/17 | cluster network (192.168.0.0 – 192.168.127.254), which allows for a total of 127 “/24” subnets, equaling 127 nodes with 254 IPs available to containers on each node. |
| 192.168.128.0/18 | portal/service network (192.168.128.0-192.168.191.254) with a total of 16382 IPs for services. |
| 192.168.192.0/24 | docker network, giving me a total of 127 IPs if I use openshift_docker_options="--bip 192.168.192.1/24 --fixed-cidr 192.168.192.1/25". This is only important for building images that have external dependencies, and doesn’t need to be large. |
Reasons why you might need to customize the OpenShift SDN
To reiterate, here are a few reasons why you might want to customize the default OpenShift network:
- The default OpenShift and docker networks created at install time conflict with the organization’s services running in the routable network range.
- Network limitations mean it is not possible to dedicate the recommended single “/14” network for the cluster and two “/16” networks for OpenShift services and docker.
Resulting configurations
As a result, the following configurations reflect the smaller networks we have defined.
| Inventory parameter | value |
| osm_cluster_network_cidr | 192.168.0.0/17 |
| osm_host_subnet_length | 8 |
| openshift_portal_net | 192.168.128.0/18 |
| openshift_docker_options | "--bip 192.168.192.1/18 --fixed-cidr 192.168.192.1/19" |
Conclusion
In this article, I have highlighted the OpenShift 3.11 inventory variables that determine how OpenShift’s SDN is created when OpenShift is installed. I have also identified some of the reasons why you would consider future-proofing your cluster for potential growth and provided an example case for customizing your SDN to mitigate the risk of IP conflict for services in your routed network and your OpenShift’s SDN.
Credits
Special thanks to Thomas Stockwell for his peer review.
References
- Red Hat OpenShift's SDN architectural overview
- Configuring Red Hat OpenShift's SDN
- Configure docker's default bridge network
- Red Hat's recommendation for modifying the docker network
- IP calculator/IP subnetting (tool I used)
- OpenShift configuration parameters: osm_cluster_network_cidr, openshift_portal_net, osm_host_subnet_length, openshift_docker_options
