Red Hat OpenShift autoscaling using MachineSet autoscaler with KEDA

There are different solutions for scaling your Red Hat OpenShift compute infrastructure. In our previous article, we demonstrated Cluster Autoscaler, the built-in, Kubernetes-native approach. In this article, we look at the MachineSet Autoscaler with KEDA, a metrics-driven approach that scales individual MachineSets based on external or custom signals.

The Custom Metrics Autoscaler operator (CMA) is Red Hat OpenShift's supported distribution of KEDA. It allows you to scale workloads based on custom metrics, like platform PromQL queries, so you don't have to wait for pods to sit in a pending state.

For MachineSet scaling, the pattern is:

1. Install the CMA operator (KEDA controller in openshift-keda).
2. Grant KEDA permission to patch machinesets/scale and read Thanos/Prometheus.
3. Create a ScaledObject targeting the MachineSet with a Prometheus trigger.
4. Ensure that no MachineAutoscaler exists on the same MachineSet (Cluster Autoscaler and KEDA must not compete).

We're using demo-p4p95-worker-eastus3 because it is the same target used in our first article about Cluster Autoscaler. Both parts scale eastus3 from 1 to 3 with the same workload, enabling a direct comparison of the 2 approaches. The MachineSet is tainted so only the scale-test workload schedules on these nodes. KEDA and Cluster Autoscaler must not run simultaneously on the same MachineSet.

Red Hat documents CMA primarily for pod workloads (deployment, StatefulSet, CRDs with pod templates). Scaling MachineSets using KEDA is demonstrated here as a metrics-driven alternative to Cluster Autoscaler. No custom horizontalPodAutoscalerConfig is required.

We don't use metricType: Value because with Value, the HPA formula factors in currentReplicas automatically:

desiredReplicas = ceil(currentReplicas × metric / threshold)

This would allow a plain utilization% query without the × nodeCount trick. However, metricType: Value relies on the HPA controller resolving "ready pods" for the scale target. Because a MachineSet has no pods, the HPA fails with unable to calculate ready pods: no pods returned by selector. This is a limitation of the HPA external-metrics path when the target is not a pod-based workload. We use metricType: AverageValue:

desiredReplicas = ceil(metric / threshold)

We compensate by embedding the replica factor directly in the PromQL query.

Manifests

First, install the Custom Metrics Autoscaler operator. Edit subscription.yaml with the namespace, OperatorGroup, and Subscription:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-keda
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-keda
  namespace: openshift-keda
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-custom-metrics-autoscaler-operator
  namespace: openshift-keda
spec:
  channel: stable
  name: openshift-custom-metrics-autoscaler-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

Apply it with kubectl:

kubectl apply -f subscription.yaml
kubectl wait --for=condition=Available deployment/keda-operator -n openshift-keda --timeout=300s

This has installed KEDA 2.19.0 (keda-operator, keda-metrics-apiserver, and keda-admission).

Adjust RBAC to allow KEDA to scale MachineSets

Edit keda-machineset-scaler-rbac.yaml to allow KEDA to scale the MachineSets:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: keda-machineset-scaler
rules:
  - apiGroups: ["machine.openshift.io"]
    resources: ["machinesets", "machinesets/scale"]
    verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-machineset-scaler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: keda-machineset-scaler
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: openshift-keda

Apply the configuration:

kubectl apply -f keda-machineset-scaler-rbac.yaml

Configure Prometheus authentication

Edit prometheus-cluster-auth-rbac.yaml with the ServiceAccount, monitoring RBAC, token Secret, and ClusterTriggerAuthentication:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: keda-thanos
  namespace: openshift-keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-thanos-monitoring-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-monitoring-view
subjects:
  - kind: ServiceAccount
    name: keda-thanos
    namespace: openshift-keda
---
apiVersion: v1
kind: Secret
metadata:
  name: keda-prom-bearer
  namespace: openshift-keda
  annotations:
    kubernetes.io/service-account.name: keda-thanos
type: kubernetes.io/service-account-token
---
apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-trigger-auth-prometheus
spec:
  secretTargetRef:
    - parameter: bearerToken
      name: keda-prom-bearer
      key: token

The kubernetes.io/service-account-token secret is populated automatically by the control plane with a bearer token in the token data key.

kubectl apply -f prometheus-cluster-auth-rbac.yaml

Verify that the token is populated (it usually only takes a few seconds):

kubectl get secret keda-prom-bearer -n openshift-keda \
-o jsonpath='{.data.token}' | \
base64 -d | head -c 20; echo

The Thanos URL is https://thanos-querier.openshift-monitoring.svc.cluster.local:9091.

Test namespace and scale-test workload

For consistency, we use the same test namespace.yaml and scale-test-deployment.yaml as in the first article. The eastus3 MachineSet is tainted (machineset-autoscaler/demo=eastus3:NoSchedule), and the deployment includes a matching toleration so only this workload schedules on eastus3 workers.

kubectl apply -f namespace.yaml
kubectl apply -f scale-test-deployment.yaml

With minReplicaCount: 1, the MachineSet always has at least one node. When the test workload fills that node beyond 75%, KEDA scales up incrementally.

Deploy the ScaledObject

Edit scaledobject-worker-eastus3.yaml to scale demo-p4p95-worker-eastus3 when global CPU request utilization across all nodes in the set exceeds 75%:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: demo-p4p95-worker-eastus3-keda
  namespace: openshift-machine-api
spec:
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: demo-p4p95-worker-eastus3
  pollingInterval: 30
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 3
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
        query: |
          sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{node=~"demo-p4p95-worker-eastus3-.*"})
          / sum(kube_node_status_allocatable{resource="cpu", node=~"demo-p4p95-worker-eastus3-.*"})
          * 100
          * count(kube_node_status_allocatable{resource="cpu", node=~"demo-p4p95-worker-eastus3-.*"})
        threshold: '75'
        activationThreshold: '0'
        authModes: bearer
        namespace: openshift-machine-api
      authenticationRef:
        name: keda-trigger-auth-prometheus
        kind: ClusterTriggerAuthentication

Apply the changes:

kubectl apply -f scaledobject-worker-eastus3.yaml
kubectl get scaledobject,hpa -n openshift-machine-api

KEDA creates an HPA (keda-hpa-demo-p4p95-worker-eastus3-keda) automatically. No custom horizontalPodAutoscalerConfig required.

CPU utilization formula

The goal: Scale the MachineSet by one replica each time global CPU utilization across all workers in the set exceeds 75%.

The query:

sum(cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests{node=~"demo-p4p95-worker-eastus3-.*"})
/ sum(kube_node_status_allocatable{resource="cpu", node=~"demo-p4p95-worker-eastus3-.*"})
* 100
* count(kube_node_status_allocatable{resource="cpu", node=~"demo-p4p95-worker-eastus3-.*"})

Breaking it down:

metric = utilization% × nodeCount
       = (sum(requests) / sum(allocatable) × 100) × count(nodes)

Why multiply by node count?

A plain utilization percentage (0–100%) is bounded, the HPA formula ceil(metric / threshold) with threshold 75 can produce at most ceil(100/75) = 2. By multiplying by the current number of nodes, the metric grows proportionally as capacity fills, enabling step-by-step scaling:

Each time utilization exceeds 75% on the current set of nodes, KEDA adds one replica.

Important terminology

Here are some terms that are important to understand.

Why we use "allocatable": The scheduler places pods against allocatable, not raw capacity. Using allocatable aligns the metric with scheduling pressure.

Why CPU requests and not usage: Requests are what the scheduler accounts for capacity, and this is consistent with Cluster Autoscaler's utilization view.

Verification

With the keda-scale-test deployment (9 replicas × 1 CPU, zone eastus-3) and minReplicaCount: 1, there are some significant results to notice.

Metric progression observed during scale-up

Each time utilization crossed 75% on the current nodes, KEDA added one replica.

KEDA ScaledObject active

NAME                             READY   ACTIVE   TRIGGERS
demo-p4p95-worker-eastus3-keda   True    True     prometheus

HPA scales the MachineSet step by step

NAME                                      TARGETS           REPLICAS
keda-hpa-demo-p4p95-worker-eastus3-keda   98829m/75 (avg)   3

MachineSet demo-p4p95-worker-eastus3 went 1 → 2 → 3, and Microsoft Azure provisioned 2 additional Standard_D4s_v3 VMs in zone 3. All 9 test pods reached the Running state (3 per node).

Timeline

00:02:51  ScaledObject applied (1 node baseline, utilization 26%)
00:03:12  Test pods deployed — 3 fit on existing node, 6 Pending
00:03:49  Metric jumps to 98.8 (99% × 1 node) — exceeds threshold 75
00:04:07  KEDA scales MachineSet 1 → 2
00:10:43  2nd node ready — 3 more pods scheduled (6 Running, 3 Pending)
00:11:01  Metric jumps to 197.7 (99% × 2 nodes) — exceeds threshold 75
00:11:01  KEDA scales MachineSet 2 → 3
00:17:35  3rd node ready — last 3 pods scheduled
00:17:53  All 9 pods Running — scale-up complete

Summary

In summary, here are the specifics of the MachineSet with KEDA autoscaling demonstration:

• Target: demo-p4p95-worker-eastus3 (min 1, max 3)
• Trigger: Prometheus (utilization% × nodeCount)
• Threshold: 75 (adds a replica each time global utilization exceeds 75%)
• Formula: ceil(util% × nodes / 75) = desired replicas
• Test load: test/keda-scale-test with 9 pods × 1 CPU, zone eastus-3, taint toleration
• Scaling: 1 → 2 → 3 step-by-step, one replica per threshold breach
• Proactive: Yes, it scales on metrics before all pods are Running
• Operator: Custom Metrics Autoscaler (KEDA 2.19)
• Conflicts: Must not share MachineSet with MachineAutoscaler
• HPA config: KEDA defaults only (no horizontalPodAutoscalerConfig)

In the next article, we compare Cluster Autoscaler as demonstrated in our first article to KEDA MachineSet.