This article demonstrates how to split an existing Red Hat OpenShift machine config pool (MCP) into two separate MCPs without requiring a reboot of any nodes. The goal is for the new MCP to have the exact same machine configuration as the original MCP.
Initially, many OpenShift clusters were built with one or two MCPs for the whole cluster. This is good when there are only 10 or 20 worker nodes. But when there are 100+ worker nodes in one MCP, this can make it difficult to perform an upgrade. As discussed in the Red Hat OpenShift Container Platform upgrade documentation, it is easier to control which nodes and how many nodes reboot if you utilize MCPs by pausing and unpausing them at the end of the upgrade.
The splitting procedure
This section outlines the prerequisites required before initiating the splitting procedure. Then it will walk you through the step-by-step procedure for splitting an OpenShift machine config pool without rebooting nodes.
Prerequisites:
- Access to OpenShift cluster with cluster-admin privileges.
- The oc CLI tool, configured and connected to the OpenShift cluster.
- An understanding of machine config pools and machine configs.
Step 1: Identify the current machine config pool
Use the following command to identify the current MCP that needs to be split:
oc get mcpExample output:
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-a5f35b903c8ad7c2c0175023f9909b05 True False False 3 3 3 0 13d
mcp-1 rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990 True False False 5 5 5 0 13d
worker rendered-worker-f19092f2278430cc4d2f9909b05a5f3 True False False 0 0 0 0 13d
Take note of two things in the output. First, there is only one MCP in this deployment that is not standard in an OpenShift Container Platform deployment, and that is “mcp-1.” Second, in the CONFIG column, there are specific hashes for each rendered version of a MCP.
Step 2: Identify the machine configs
Identify the machine configs by retrieving the machine configs associated with the current MCP using the following:
oc get mcp mcp-1 -o json | jq .spec.configurationExample output:
{
"name": "rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990",
"source": [
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "00-worker"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "01-worker-container-runtime"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "01-worker-kubelet"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "50-workers-chrony-configuration"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "97-worker-generated-kubelet"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "98-worker-generated-kubelet"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "99-worker-generated-registries"
},
{
"apiVersion": "machineconfiguration.openshift.io/v1",
"kind": "MachineConfig",
"name": "99-worker-ssh"
}
]
}This will display each MachineConfig applied to this specific MCP.
If you are not sure of the specifics regarding each MachineConfig, you can use the following command:
oc get mcExample output:
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d 00-worker 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d 01-master-container-runtime 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
01-master-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
01-worker-container-runtime 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
01-worker-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
50-masters-chrony-configuration 3.1.0 13d
50-workers-chrony-configuration 3.1.0 13d
97-master-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
97-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
97-worker-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
98-master-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
98-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
98-worker-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
99-assisted-installer-master-ssh 3.1.0 13d
99-master-generated-registries 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
99-master-ssh 3.2.0 13d
99-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 4d5h
99-worker-generated-registries 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
99-worker-ssh 3.2.0 13d
rendered-master-a5f35b903c8ad7c2c0175023f9909b05 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
rendered-mcp-1-8f6a90f60f8d9db7d8c0e243c9bf4963 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
rendered-mcp-1-b9d21061d076437680290f5da831ada0 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 24h
rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
rendered-mcp-1-f90b4a73043e3cec10a72075c4d9d9fb 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 4d5h
rendered-worker-dcb907f73ccf964e6db1cf7db6cd45ab 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
rendered-worker-f19092f2278430cc4d2cff6d9400f990 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 13d
If you need any further assistance regarding MachineConfig, please refer to the OpenShift Container Platform documentation.
Step 3: Create a new machine config pool
Create a new MCP definition YAML file, mcp-new.yaml, with the desired name for the new pool (e.g., mcp-new). The machineConfigSelector for this new MCP should be exactly the same as the original MCP, as in this example:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: mcp-new
spec:
machineConfigSelector:
matchExpressions:
- {
key: machineconfiguration.openshift.io/role,
operator: In,
values: [worker,mcp-1]
}
nodeSelector:
matchLabels:
node-role.kubernetes.io/mcp-new: "" # Label for nodes to move to this pool (nodes to be updated later)Apply the new MCP as follows:
oc apply -f mcp-new.yamlStep 4: Verify both MCPs have the same rendered hash
To make sure that nodes don’t reboot when they move from one MCP to another, the hash on both MCPs has to be the same. This means that MachineConfigs on both MCPs are exactly the same. Therefore, the Red Hat Enterprise Linux CoreOS configuration within these MCPs won’t require a change to the OS or files on the host when moved.
To verify, run the following command:
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-a5f35b903c8ad7c2c0175023f9909b05 True False False 3 3 3 0 14d
mcp-1 rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990 True False False 5 5 5 0 14d
mcp-new rendered-mcp-new-f19092f2278430cc4d2cff6d9400f990 True False False 0 0 0 0 42h
worker rendered-worker-f19092f2278430cc4d2cff6d9400f990 True False False 0 0 0 0 14dStep 5: Label the nodes
In this step, we will label the nodes for the new machine config pool. Identify the nodes moving to the new MCP and label them with the node selector label defined in the new MCP’s nodeSelector.
# oc get node
NAME STATUS ROLES AGE VERSION
ctrl-plane-0.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
ctrl-plane-1.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
ctrl-plane-2.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
worker-0.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-1.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-2.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-3.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-5.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387To move nodes from mcp-1 to mcp-new, patch the desired node(s) with the following command:
# oc patch node worker-5.nantahala.daytwops.bos2.lab --type=json -p '[{"op":"move","from": "/metadata/labels/node-role.kubernetes.io~1mcp-1", "path": "/metadata/labels/node-role.kubernetes.io~1mcp-new"}]'Note: In the patch command, the ~1 is how the “/” character escaped. In this example, it allows us to reference a label which contains a “/” character.
Run the following command to watch the change:
# watch "oc get no; echo; oc get mcp; echo; oc get mc| grep -vE 'worker|master'"Example output:
NAME STATUS ROLES AGE VERSION
ctrl-plane-0.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
ctrl-plane-1.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
ctrl-plane-2.nantahala.daytwops.bos2.lab Ready control-plane,master 14d v1.29.10+67d3387
worker-0.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-1.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-2.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-3.nantahala.daytwops.bos2.lab Ready mcp-1,worker 14d v1.29.10+67d3387
worker-5.nantahala.daytwops.bos2.lab Ready mcp-new,worker 14d v1.29.10+67d3387
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-a5f35b903c8ad7c2c0175023f9909b05 True False False 3 3 3 0 14d
mcp-1 rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990 True False False 4 4 4 0 14d
mcp-new rendered-mcp-new-f19092f2278430cc4d2cff6d9400f990 True False False 1 1 1 0 42h
worker rendered-worker-f19092f2278430cc4d2cff6d9400f990 True False False 0 0 0 0 14d
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
97-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
97-mcp-new-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
98-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
98-mcp-new-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
99-mcp-1-generated-kubelet 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 5d
rendered-mcp-1-8f6a90f60f8d9db7d8c0e243c9bf4963 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
rendered-mcp-1-b9d21061d076437680290f5da831ada0 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 43h
rendered-mcp-1-f19092f2278430cc4d2cff6d9400f990 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
rendered-mcp-1-f90b4a73043e3cec10a72075c4d9d9fb 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 5d
rendered-mcp-new-8f6a90f60f8d9db7d8c0e243c9bf4963 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
rendered-mcp-new-f19092f2278430cc4d2cff6d9400f990 64460169e27091d8f8373b0952604ba2700d6d67 3.4.0 14d
...You will see the mcp-new go into updated = false and updating = true. Then it will immediately change back to updated = true and updating = false when the node moves into mcp-new. You will also see the MACHINECOUNT change.
You should not see the node go into SchedulingDisabled. This would indicate that the configuration between these two MCPs was incorrect, and that node will reboot.
Important to note
Ensure the machineConfigSelector for the new MCP exactly matches the original to avoid unintended configuration changes and potential disruptions. Patching node labels is the action that triggers the move to a new MCP.
Monitor node status and MCP status closely during and after the process. Plan and coordinate this operation during a maintenance window, even if you don’t expect a reboot.
Summary
This procedure offers a significant advantage by allowing the splitting of a machine config pool without triggering node reboots, provided the machine configs remain identical between the old and new pools. This ensures a seamless transition and avoids service disruption, making upgrades and cluster management more efficient.
