Red Hat OpenShift has become the application platform of choice for many customers who need to onboard applications on a daily basis. As demand increases, it is important to make sure the networks within OpenShift are reliable and easy to handle. OpenShift 4.19 and later supports Border Gateway Protocol (BGP) routing through Free Range Routing (FRR), a free open source internet routing protocol for Linux, UNIX, and other similar platforms.
Within the OpenShift Cluster Network Operator, when you enable FRR parameters, the Cluster Network Operator runs an FRR daemonset with one pod replica on every node in a dedicated namespace, openshift-frr-k8s. As a platform administrator, you can configure the FRRConfiguration custom resource to access this service and expose internal networks to external subsystems.
In this section, we will look at how to use the Kubernetes custom resource to expose selective OpenShift networks to external network tools. Finally, we show a test of a pod with a different network scheduled on a specific OpenShift node and how it accesses servers outside its network.
OpenShift networks and BGP configuration enablement
As organizations adopt cloud-native models and hybrid deployments, their networking needs become more sophisticated. Solutions like Red Hat OpenShift depend on resilient, adaptable, and high-performance networking. Integrating BGP with supporting technologies, such as BFD and cluster user-defined networks (UDNs), is particularly effective because it delivers:
- Improved workload connectivity through dynamic routing and real-time route visibility
- Better traffic isolation and flexible network segmentation
- Rapid network convergence for virtual machine (VM) traffic
Prerequisites
Make sure the following prerequisites are met before starting the BGP configuration on OpenShift.
Ensure your cluster is running OpenShift 4.20 or later.
To activate the extra routing features on our OpenShift cluster, apply this patch to the network operator:
oc patch Network.operator.openshift.io cluster --type=merge -p='{"spec":{"additionalRoutingCapabilities":{"providers":["FRR"]},"defaultNetwork":{"ovnKubernetesConfig":{"routeAdvertisements":"Enabled"}}}}'This command enables the FRR routing provider and turns on OVN-Kubernetes route advertisements in the cluster operator configuration.
[root@vm-236-67 ~]# oc get network.operator -o yaml | egrep -i "frr|route" -B2
additionalRoutingCapabilities:
providers:
- FRR
--
routeAdvertisements: Enabled
[root@vm-236-67 ~]# oc get pods -n openshift-frr-k8s
NAME READY STATUS RESTARTS AGE
frr-k8s-22v8d 7/7 Running 0 64s
frr-k8s-2trmd 7/7 Running 0 63s
frr-k8s-6ngvl 7/7 Running 0 64s
frr-k8s-7qj9c 7/7 Running 0 64s
frr-k8s-8xtts 7/7 Running 0 64s
frr-k8s-bvfgk 7/7 Running 0 64s
frr-k8s-gtv4s 7/7 Running 0 63s
frr-k8s-lvbgq 7/7 Running 0 64s
frr-k8s-mv9q8 7/7 Running 0 63s
frr-k8s-szztb 7/7 Running 0 63s
frr-k8s-webhook-server-dcc49db58-vpqb6 1/1 Running 0 63s
frr-k8s-wx7g2 7/7 Running 0 64s
frr-k8s-xbxxr 7/7 Running 0 64s
frr-k8s-xp2d4 7/7 Running 0 64s
frr-k8s-xz4gk 7/7 Running 0 63sThe above section shows the daemonset running on every node in a new namespace now created and managed by a cluster network operator. If your cluster is running MetalLB in BGP mode, you can migrate the same configuration to a existing namespace
FRRConfiguration with selective labels for Networks
The FRRConfiguration is a Kubernetes custom resource definition (CRD) that lets you configure FRRouting (FRR) instances for BGP routing. It specifies and controls BGP routing settings on cluster nodes to enable automated and adaptive network routing for workloads like containers and VMs.
To define the FRRConfiguration resource in OpenShift, we specify 64512 as the Autonomous System (AS) number and list the IP address of the external BGP router as a BGP neighbor.
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: default
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 64512
neighbors:
- address: 192.168.236.67
asn: 65000
disableMP: true
dualStackAddressFamily: false
toAdvertise:
allowed:
mode: all
toReceive:
allowed:
mode: all
nodeSelector:
matchLabels:
network: secondaryThis FRR configuration recognizes the neighbor on AS 65000. Using the nodeSelector component, we could select a specific set of nodes to announce the pod network to neighbors.
OpenShift is a shared platform for different workloads. Because you can select certain node networks to expose to external routers, you should label some nodes on the cluster with network=secondary.
oc label node worker0.example.com worker1.example.com worker3.example.com worker4.example.com network=secondaryAfter you label the nodes, the Cluster Network Operator learns to expose only those networks to the external BGP neighbor based on the default FRRConfiguration given to the cluster.
Along with the pod network, users can also host selective cluster user-defined networks (CUDN) to the external BGP router when the VMs using the CUDN require dynamic routing.
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: default
spec:
frrConfigurationSelector: {}
networkSelectors:
- clusterUserDefinedNetworkSelector:
networkSelector:
matchLabels:
advertise: "true"
networkSelectionType: ClusterUserDefinedNetworks
nodeSelector: {}
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
labels:
advertise: "true"
name: extranet
spec:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- udn-demo-l3-ns1
- udn-demo-l3-ns2
network:
layer2:
role: Primary
subnets:
- 22.100.0.0/16
topology: Layer2The RouteAdvertisements resource ensures the cluster advertises its CUDN network matching the advertise=true label. This controls the advertisement of selective networks, while the FRRConfiguration identifies the neighbor and selects the participating nodes.
$ oc get frrconfigurations.frrk8s.metallb.io
NAME AGE
default 2m48s
ovnk-generated-575hg 2m47s
ovnk-generated-92497 2m47s
ovnk-generated-ls7rp 2m47s
ovnk-generated-thxxb 2m47s
ovnk-generated-v7rzs 2m48s
ovnk-generated-ww9fp 2m47sAfter the RouteAdvertisement advertises the pod networks, use the FRRConfiguration to control which nodes act as BGP routers. This ensures only specific node networks are exposed to the external BGP router.
Confirming the configuration on the BGP router
Let's look at how the Cluster Network Operator presents the configuration after reading the RouteAdvertisment and the default FRRConfiguration.
An FRRConfiguration object is generated for each network and node selected by a RouteAdvertisements custom resource (CR) with the appropriate advertised prefixes that apply to each node. The OVN-Kubernetes controller verifies that nodes selected by the RouteAdvertisements CR are a subset of the nodes selected by the FRRConfiguration.
$ oc get frrnodestates.frrk8s.metallb.io worker5.example.com -o yaml
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRNodeState
metadata:
generation: 1
name: worker5.example.com
spec: {}
status:
lastConversionResult: success
lastReloadResult: success
runningConfig: |
Building configuration...
Current configuration:
!
frr version 8.5.3
frr defaults traditional
hostname worker5.example.com
log file /etc/frr/frr.log informational
log timestamp precision 3
service integrated-vtysh-config
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
end
Now let's look at the frrnodestate configuration for a node that matches the nodeSelector specified in the FRRConfiguration.
$ oc get frrnodestates worker4.example.com -o yaml
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRNodeState
metadata:
name: worker4.example.com
spec: {}
status:
lastConversionResult: success
lastReloadResult: success
runningConfig: |
Building configuration...
Current configuration:
!
frr version 8.5.3
frr defaults traditional
hostname worker4.example.com
log file /etc/frr/frr.log informational
log timestamp precision 3
service integrated-vtysh-config
!
router bgp 64512
no bgp ebgp-requires-policy
no bgp hard-administrative-reset
no bgp default ipv4-unicast
no bgp graceful-restart notification
bgp graceful-restart preserve-fw-state
no bgp network import-check
neighbor 192.168.236.67 remote-as 65000
!
address-family ipv4 unicast
network 10.131.4.0/23
network 22.100.0.0/16
neighbor 192.168.236.67 activate
neighbor 192.168.236.67 route-map 192.168.236.67-in in
neighbor 192.168.236.67 route-map 192.168.236.67-out out
import vrf extranet
exit-address-family
!
address-family ipv6 unicast
import vrf extranet
exit-address-family
exit
!
router bgp 64512 vrf extranet
no bgp ebgp-requires-policy
no bgp hard-administrative-reset
no bgp default ipv4-unicast
no bgp graceful-restart notification
bgp graceful-restart preserve-fw-state
no bgp network import-check
!
address-family ipv4 unicast
import vrf default
exit-address-family
!
address-family ipv6 unicast
import vrf default
exit-address-family
exit
!
ip prefix-list 192.168.236.67-allowed-ipv4 seq 1 permit 10.131.4.0/23
ip prefix-list 192.168.236.67-allowed-ipv4 seq 2 permit 22.100.0.0/16
ip prefix-list 192.168.236.67-inpl-ipv4 seq 1 permit any
!
ipv6 prefix-list 192.168.236.67-allowed-ipv6 seq 1 deny any
ipv6 prefix-list 192.168.236.67-inpl-ipv4 seq 2 permit any
!
route-map 192.168.236.67-out permit 1
match ip address prefix-list 192.168.236.67-allowed-ipv4
exit
!
route-map 192.168.236.67-out permit 2
match ipv6 address prefix-list 192.168.236.67-allowed-ipv6
exit
!
route-map 192.168.236.67-in permit 3
match ip address prefix-list 192.168.236.67-inpl-ipv4
exit
!
route-map 192.168.236.67-in permit 4
match ipv6 address prefix-list 192.168.236.67-inpl-ipv4
exit
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
endNext, check the external BGP router configuration. It should receive four pod networks and one CUDN from the OpenShift cluster.
[root@frr-router /]# vtysh
Hello, this is FRRouting (version 8.5.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
frr-router# show bgp all
For address family: IPv4 Unicast
BGP table version is 24, local router ID is 192.168.236.67, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.128.2.0/23 192.167.105.243 0 0 64512 i
*> 10.128.6.0/23 192.167.104.214 0 0 64512 i
*> 10.129.6.0/23 192.167.103.189 0 0 64512 i
*> 10.131.4.0/23 192.167.104.51 0 0 64512 i
*= 22.100.0.0/16 192.167.103.189 0 0 64512 i
*= 192.167.104.51 0 0 64512 i
*= 192.167.104.214 0 0 64512 i
*> 192.167.105.243 0 0 64512 i
*> 12.10.2.0/23 0.0.0.0 0 32768 ?
Displayed 6 routes and 9 total paths
[root@frr-router /]# ip r
default via 192.167.109.254 dev ens192 proto dhcp src 192.168.236.67 metric 100
default via 192.168.3.254 dev ens224 proto dhcp src 192.168.2.72 metric 101
192.167.102.0/21 dev ens192 proto kernel scope link src 192.168.236.67 metric 100
10.88.0.0/16 dev podman0 proto kernel scope link src 10.88.0.1
10.128.2.0/23 via 192.167.105.243 dev ens192 proto bgp metric 20
10.128.6.0/23 via 192.167.104.214 dev ens192 proto bgp metric 20
10.129.6.0/23 via 192.167.103.189 dev ens192 proto bgp metric 20
10.131.4.0/23 via 192.167.104.51 dev ens192 proto bgp metric 20
22.100.0.0/16 proto bgp metric 20
nexthop via 192.167.103.189 dev ens192 weight 1
nexthop via 192.167.104.51 dev ens192 weight 1
nexthop via 192.167.104.214 dev ens192 weight 1
nexthop via 192.167.105.243 dev ens192 weight 1
192.168.2.0/23 dev ens224 proto kernel scope link src 192.168.2.72 metric 101
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdowTesting pod network access from provider network
Now that our external BGP router knows about our pod network and the OpenShift cluster knows about the BGP router, let's test connectivity from a container running on an exposed OpenShift node.
[root@vm-236-67 ~]# oc get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-6db48b7596-v8ts2 1/1 Running 0 14s 10.131.4.157 worker4.example.com <none> <none>
virt-launcher-fedora-maroon-swan-24-dtcwt 2/2 Running 0 41h 10.131.4.156 worker4.example.com <none> 1/1
$ oc rsh nginx-deployment-6db48b7596-v8ts2
sh-4.4$ ping -c 3 192.168.236.67
PING 192.168.236.67 (192.168.236.67) 56(84) bytes of data.
64 bytes from 192.168.236.67: icmp_seq=1 ttl=63 time=3.21 ms
64 bytes from 192.168.236.67: icmp_seq=2 ttl=63 time=1.34 ms
64 bytes from 192.168.236.67: icmp_seq=3 ttl=63 time=0.832 ms
--- 192.168.236.67 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.832/1.795/3.214/1.024 ms
sh-4.4$ ip r
default via 22.100.0.1 dev ovn-udn1
10.128.0.0/14 via 10.131.4.1 dev eth0
10.131.4.0/23 dev eth0 proto kernel scope link src 10.131.4.157
22.100.0.0/16 dev ovn-udn1 proto kernel scope link src 22.100.0.13
100.64.0.0/16 via 10.131.4.1 dev eth0
100.65.0.0/16 via 22.100.0.1 dev ovn-udn1
172.30.0.0/16 via 22.100.0.1 dev ovn-udn1Wrap up
When route advertisements are enabled and an FRRConfiguration identifies the OpenShift neighbor, the OVN-Kubernetes network plug-in can announce routes for the default pod network and CUDNs for the provider network, including egress IPs. The plug-in also supports importing routes from the provider network into the default pod network and CUDNs. As a result, IP addresses advertised from the default pod network and CUDNs are directly reachable from the provider network.
