Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

How to customize the Red Hat OpenShift 3.11 SDN

November 1, 2019
Mohammad Ahmad
Related topics:
Kubernetes
Related products:
Red Hat OpenShiftRed Hat OpenShift Container Platform

Share:

    In this article, I'll highlight a practical case for customizing the Red Hat OpenShift software-defined network (SDN). To achieve this, I will identify the OpenShift-Ansible inventory parameters that configure different aspects of the OpenShift SDN, specifically the cluster, portal, and docker networks.

    Why customize the SDN?

    An important question I am often asked is: Why do you need to customize the SDN? Isn’t it completely internal? Users generally assume there is no need to customize the SDN because OpenShift’s SDN has no impact on networks outside the OpenShift cluster; therefore, IP conflicts should not be a concern. However, this is not always the case.

    In large organizations, designated private networks (e.g., 10.0.0.0/8) are often used “publicly” within the internal confines of the organization's own network. With that in mind, the primary concern is that services external to the OpenShift cluster may be using IPs within the normally private ranges that conflict with those used by the cluster itself.

    These conflicts will cause issues when applications in the OpenShift cluster attempt to communicate to these conflicting external services, as the OpenShift SDN will attempt to route these requests within its own network, causing these services to be unreachable from within the cluster.

    Background

    Default values

    The default networks configured by OpenShift at installation time are:

    Component Ansible inventory parameters Notes
    The cluster network osm_cluster_network_cidr and osm_host_subnet_length which by default are 10.128.0.0/14 and 9 respectively.

     

     

    Each node gets a “/23” subnet from this range, allowing a total of 512 subnets dedicated for nodes, and a total of 510 IPs on each node for containers.
    Running “oc get pods --all-namespaces -o wide” will show which IPs have been allocated from this range, and should all be within this range.
    The services network openshift_portal_net By default this is 172.30.0.0/16. This means you have a total of 65534 addresses to assign to services.

     

    Running “oc get services --all-namespaces” should also show that all service IPs are within this range.

    The docker network openshift_docker_options="--bip 172.17.0.1/16 --fixed-cidr 172.17.0.0/17" These values are command line arguments to the docker daemon configured by the openshift_docker_options variable in openshift-ansible.

    Customizing these values

    Fine-tuning each of these values dictates how large your cluster can grow by way of defining how many nodes, pods, and services it can handle. With that in mind, for a development cluster, you may want to allow more services, as opposed to pods, to address specific needs required by developers. However, on a production system, you may prefer the ability to scale in the number of nodes to address a business and operational requirement.

    This article is based on the situation where you only have the subnet 192.168.0.0/16 for the entire cluster to use. With that in mind, I partition the subnet as follows:

    Subnet Use
    192.168.0.0/17 cluster network (192.168.0.0 – 192.168.127.254), which allows for a total of 127 “/24” subnets, equaling 127 nodes with 254 IPs available to containers on each node.
    192.168.128.0/18 portal/service network (192.168.128.0-192.168.191.254) with a total of 16382 IPs for services.
    192.168.192.0/24 docker network, giving me a total of 127 IPs if I use openshift_docker_options="--bip 192.168.192.1/24 --fixed-cidr 192.168.192.1/25". This is only important for building images that have external dependencies, and doesn’t need to be large.

    Reasons why you might need to customize the OpenShift SDN

    To reiterate, here are a few reasons why you might want to customize the default OpenShift network:

    • The default OpenShift and docker networks created at install time conflict with the organization’s services running in the routable network range.
    • Network limitations mean it is not possible to dedicate the recommended single “/14” network for the cluster and two “/16” networks for OpenShift services and docker.

    Resulting configurations

    As a result, the following configurations reflect the smaller networks we have defined.

    Inventory parameter value
    osm_cluster_network_cidr 192.168.0.0/17
    osm_host_subnet_length 8
    openshift_portal_net 192.168.128.0/18
    openshift_docker_options "--bip 192.168.192.1/18 --fixed-cidr 192.168.192.1/19"

    Conclusion

    In this article, I have highlighted the OpenShift 3.11 inventory variables that determine how OpenShift’s SDN is created when OpenShift is installed. I have also identified some of the reasons why you would consider future-proofing your cluster for potential growth and provided an example case for customizing your SDN to mitigate the risk of IP conflict for services in your routed network and your OpenShift’s SDN.

    Credits

    Special thanks to Thomas Stockwell for his peer review.

    References

    • Red Hat OpenShift's SDN architectural overview
    • Configuring Red Hat OpenShift's SDN
    • Configure docker's default bridge network
    • Red Hat's recommendation for modifying the docker network
    • IP calculator/IP subnetting (tool I used)
    • OpenShift configuration parameters: osm_cluster_network_cidr, openshift_portal_net, osm_host_subnet_length, openshift_docker_options
    Last updated: January 12, 2024

    Recent Posts

    • How Kafka improves agentic AI

    • How to use service mesh to improve AI model security

    • How to run AI models in cloud development environments

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue