Configure a cluster

Let’s dig into our successful deployment of Red Hat OpenShift Data Foundations (ODF) using availability zones. 

We completed the following path within a Red Hat lab environment on a three-node compact cluster with each node deployed in a different rack using a spine/leaf network architecture. Each leaf switch is configured with Virtual Local Area Network (VLAN) IDs 100 and 200 and Dynamic Host Configuration Protocol (DHCP) forwarding to a DHCP server.  The IP ranges for each VLAN is unique and managed by the DHCP server.  

For brevity, we assume the list of Operators below has been installed and any appropriate instances have been configured (For example, an NMState instance has been created).  We also assume admin access rights using command-line interface (CLI) commands are available.

Prerequisites:

• Kubernetes NMState operator
• Red Hat Local storage operator
• Red Hat OpenShift Data Foundation operator
• An OpenShift/Kubernetes environment 

In this lesson, you will:

• Apply network configuration policy and network attachment definition.
• Learn how to add a Network Attachment Definition (NAD) to the cluster.

Network node configuration policy

Complete the following steps after the above operators have been installed and before the ODF storage cluster has been created.  Figure 1 provides a visual representation of the network configuration that will be applied.

Network node configuration policy

The Network node configuration policy (NNCP) required to configure two of the physical network interfaces, the VLAN tagged interfaces, and the Multus ODF shim on each node is below.  Link Aggregation Control Protocol (LACP) will be configured for our bonding mode using two ports from a four-port 25Gb Ethernet card.  The two VLAN interfaces are 100 and 200 for the public and cluster ODF networks respectively.  It is worth noting the shim interface must be defined as a macvlan and be connected to the same interface as will be defined in the Network Attachment Definition for the ODF public network.  It is also using DHCP to obtain an IP address but could also be configured using a static IP address.  A shim interface is required for the public network but is not required for the cluster network.  The reason for this is the cluster network is used by the OSD pods only, with no client communication.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: bond1
spec:
  desiredState:
    interfaces:
      - description: Bond for OpenShift Data Foundation Storage Networks
        link-aggregation:
          mode: 802.3ad
          options:
            miimon: '100'
          port:
            - eno1p2
            - eno1p3
        mtu: 9000
        name: bond1
        state: up
        type: bond
      - name: bond1.100
        state: up
        type: vlan
        vlan:
          base-iface: bond1
          id: 100
      - name: bond1.200
        state: up
        type: vlan
        vlan:
          base-iface: bond1
          id: 200
      - description: Shim interface used to connect host to OpenShift Data Foundation public Multus network
        ipv4:
          dhcp: true
          enabled: true
        mac-vlan:
          base-iface: bond1.100
          mode: bridge
          promiscuous: true
        name: odf-pub-shim
        state: up
        type: mac-vlan
  nodeSelector:
    node-role.kubernetes.io/worker: ''

Apply the NNCP using the oc apply command.

oc apply -f odf-network.yaml
nodenetworkconfigurationpolicy.nmstate.io/bond1 created

Verify the NNCPs were applied successfully using the oc get command.  This may take a few minutes.  Wait until the STATUS is Available and the REASON is SuccessfullyConfigured.

oc get nncp              
NAME                  STATUS        REASON
bond1                 Available     SuccessfullyConfigured

Now that the NNCPs have been applied successfully, it’s time to apply a network attachment definition (NAD) for the private and cluster network. 

Network attachment definition

The first NAD defined on the bond1.100 interface is for the public network traffic isolation.  This is used by client applications and pods to interact with the storage cluster.  The .spec.config.type is macvlan, the .spec.config.ipam.type is set to dhcp with the .spec.config.request.option of 121.  This option specifies that the client (worker nodes in our case) should add the static routes provided by the DHCP server to its routing table.  

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: odf-public-network
  namespace: openshift-storage
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "bond1.100",
    "mode": "bridge",
    "ipam": {
      "type": "dhcp",
      "request": [
        {"option": "121"}
      ]
    }
  }'

Apply the NAD for the private network using the oc apply command.

oc apply -f odf-public-nad.yaml
networkattachmentdefinition.k8s.cni.cncf.io/odf-public-network created

A NAD for the cluster network was also defined using the bond1.vlan200 interface.  This network will be used for internal storage replication and rebalancing.  

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: odf-cluster-network
  namespace: openshift-storage
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "bond1.200",
    "mode": "bridge",
    "ipam": {
      "type": "dhcp",
      "request": [
        {"option": "121"}
      ]
    }
  }'

Apply the NAD for the cluster network using the oc apply command.

oc apply -f odf-cluster-nad.yaml
networkattachmentdefinition.k8s.cni.cncf.io/odf-cluster-network created