Declarative clusters: GitOps deployment of hosted control planes on Red Hat OpenShift Virtualization

October 6, 2026
Serena Flocco
Related topics:
VirtualizationGitOpsAutomationHelm
Related products:
Red Hat OpenShift VirtualizationRed Hat OpenShift GitOpsRed Hat Advanced Cluster Management for Kubernetes

In modern enterprise data centers running on bare-metal infrastructure, resource efficiency and agility are paramount. Traditionally, deploying a new Red Hat OpenShift cluster required dedicating at least 3 physical nodes for the control plane. When multiplying this across development, test, and production environments, the hardware footprint grows rapidly, leaving valuable compute power underutilized.

Hosted control planes for Red Hat OpenShift, architecturally based on the HyperShift project, decouples the cluster control plane from the worker nodes. By hosting the control plane as ordinary pod workloads inside a central management cluster, you can drastically reduce infrastructure costs and spin up new clusters in minutes.

But how do we scale this model sustainably? By combining hosted control planes with Red Hat OpenShift Virtualization, we can run worker nodes as virtual machines (VMs) and automate the entire lifecycle via Red Hat Advanced Cluster Management and Red Hat OpenShift GitOps, achieving a true "Cluster-as-a-Service" model.

This article provides a step-by-step guide to building a declarative, production-ready GitOps workflow using Helm and Argo CD ApplicationSets to deploy automated Red Hat OpenShift hosted control plane clusters on bare metal.

Architecture overview

The proposed solution is one of the several possible solutions you can adopt using hosted control planes and OpenShift Virtualization to operate a "Cluster-as-a-Service" model. Depending on the cluster separation and multitenancy requirements, a different architecture could be implemented, but the logic under the hood remains the same regardless of the solution you choose.

Before diving into the technical solution, it is essential to understand the hosted control planes topology and how the different components interact within our GitOps workflow (Figure 1).

GitOps-driven hosted control planes architecture
Figure 1: High-level architecture of hosted control planes on bare metal.

The architecture is composed of the following key pillars:

  • The bare-metal management cluster: a physical Red Hat OpenShift cluster deployed on bare-metal hardware. It hosts OpenShift Virtualization, Red Hat Advanced Cluster Management, and OpenShift GitOps.
  • The hosted control plane: the API server, controller manager, and etcd (plus all the additional cluster operators) for the hosted cluster run as pods inside a dedicated namespace on the management cluster.
  • The hosted worker nodes: OpenShift Virtualization provisions virtual machines on the bare-metal infrastructure. These VMs boot up, connect to the hosted control plane pods, and form the hosted cluster workload capacity.
  • The hosted cluster: the combination of the hosted control plane and the hosted worker nodes creates the hosted cluster. From a developer's perspective, this cluster looks and acts exactly like a standard standalone Red Hat OpenShift cluster, but it is provisioned in a fraction of the time and with a significantly smaller hardware footprint. From an administrative point of view, a hosted cluster acts like a standalone Red Hat OpenShift cluster, but with minor differences due to the distinct control plane setup.

By using Red Hat Advanced Cluster Management for Kubernetes, we gain a unified API to orchestrate HyperShift, while OpenShift GitOps ensures that our cluster definitions remain version-controlled and synchronized with Git.

Prerequisites

The solution explained in this article assumes the availability of a Red Hat OpenShift bare-metal cluster. See Installing OpenShift Container Platform on bare metal for a complete setup guide.

Before deploying hosted control planes via GitOps, the underlying bare-metal management cluster requires strict preparation, specifically around networking and storage.

Step 1: Advanced networking configuration

Network segmentation could help to separate different data paths. Figure 2 shows the network setup proposed in this lab. Note that, as always, there are several different options to implement network segmentation, so it is suggested to choose the one that best fits the architecture requirements.

Bare-metal network topology
Figure 2: Proposed network setup for the lab environment.

In this setup, 3 distinct networks are defined using the Kubernetes NMState Operator:

  • Management network: the default infrastructure network for the bare-metal cluster components.
  • Live migration network: a dedicated high-speed network (for example, 10/25 Gbps) exclusively for migrating worker VMs between physical nodes without saturating the management API. See Configuring a dedicated network for live migration.
  • Hosted clusters network: VLAN-encapsulated traffic passed to the physical nodes. Using NMState, Multus CNI, and network attachment definitions, these VLANs are bridged directly to the hosted control plane worker VMs. Note that in this case the VMs are not attached to the default pod network and an external DHCP service must be available to provide IP addresses to VMs.

In addition to this configuration, because bare-metal environments lack a native cloud provider load balancer, MetalLB must be installed and configured to provide the external IP address required to expose the hosted control plane API server, as shown in Figure 3.

Bare-metal network configuration
Figure 3: MetalLB load balancer and NMState networking integration.

Step 2: RWX storage configuration

Because the worker nodes are virtual machines, enabling physical node mobility (live migration) requires a storage backend capable of ReadWriteMany (RWX) access. See the storage configuration for OpenShift Virtualization guide.

  • Red Hat OpenShift Data Foundation (ODF) can provide all the features needed for live migration capability.
  • Other Red Hat-certified storage vendors are fully supported as long as the Container Storage Interface (CSI) driver provides the requested capabilities.

Step 3: Core Red Hat OpenShift operators installation

Finally, the foundation requires the installation of 3 additional operators via the OpenShift Ecosystem Catalog:

The GitOps engine: Helm and Argo CD ApplicationSet

To achieve a truly scalable "Cluster-as-a-Service" model, we rely on Helm for infrastructure templating and an Argo CD ApplicationSet to dynamically generate the deployments.

As a one-time setup, you must create the namespace on the management cluster where all hosted control plane resources will reside:

oc create namespace clusters

To pull OpenShift Container Platform release images and access VM nodes via SSH, the clusters namespace on the management cluster must contain the pull secret and the SSH key secret.

According to best practices, secrets should never be committed to Git. In a production scenario, the external secrets operator can be used to inject values dynamically.

Step 1: Git repository structure

The Git repository is organized into 3 subdirectories:

  • gitops: Contains the Argo CD resources required to bootstrap the workflow on the management cluster.
  • helm: The immutable infrastructure code. It contains the parameterized YAML templates needed to create the hosted control planes and manage the Advanced Cluster Management registration.
  • clusters: The data layer. It contains simple, tenant-specific configuration files (for example, VLANs, number of workers, OpenShift Container Platform version) that configure the Helm templates.

The complete structure is as follows:

ocp-gitops-fleet/
├── gitops/
│   ├── appproject.yaml
│   └── applicationset.yaml
├── helm/
│   ├── Chart.yaml
│   ├── values.yaml
│   └── templates/
│       ├── hostedcluster.yaml
│       ├── nodepool.yaml
│       └── acm-registration.yaml
└── clusters/
    ├── ocp-dev-01.yaml
    └── ocp-prod-01.yaml

Step 2: GitOps manifests

Inside the gitops directory, we define the core engine of our automation. While appproject.yaml simply provides a logical boundary and RBAC policies for our fleet, the true orchestrator is the applicationset.yaml.

The Argo CD ApplicationSet uses a Git file generator to watch the clusters folder: for every file it discovers, it dynamically creates a new Argo CD Application, mapping the tenant's variables directly to the Helm chart.

Here is the ApplicationSet definition:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: hcp-fleet-manager
  namespace: openshift-gitops
spec:
  generators:
    - git:
        repoURL: 'https://github.com/your-org/ocp-gitops-fleet.git'
        revision: HEAD
        files:
          - path: "clusters/*.yaml"
  template:
    metadata:
      name: '{{clusterName}}'
    spec:
      project: hcp-fleet-project
      source:
        repoURL: 'https://github.com/your-org/ocp-gitops-fleet.git'
        targetRevision: HEAD
        path: helm
        helm: 
          valueFiles:
            - ../clusters/{{clusterName}}.yaml 
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: clusters
      syncPolicy:
        automated:
          prune: false # this prevents accidental cluster deletion
          selfHeal: true

The appproject.yaml manifest is defined as follows:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: hcp-fleet-project
  namespace: openshift-gitops
spec:
  description: "Enterprise Project for Hosted Control Planes Fleet Management"
  sourceRepos:
    - 'https://github.com/your-org/ocp-gitops-fleet.git'
  destinations:
    - namespace: '*'
      server: 'https://kubernetes.default.svc'
  clusterResourceWhitelist:
    - group: 'cluster.open-cluster-management.io'
      kind: 'ManagedCluster'

Before Argo CD can automatically provision clusters based on Git commits, the GitOps manifests have to be applied to the management cluster.

Remember to assign the appropriate role-based access controls (RBAC) to the Argo CD application controller service account to avoid "permission denied" issues and configure the credentials to access the Git repository in OpenShift GitOps.

Log in to your management cluster and apply the fleet manager:

oc apply -f appproject.yaml
oc apply -f applicationset.yaml

Step 3: Helm templates

The helm directory acts as our immutable infrastructure engine.

First, a standard Chart.yaml should be defined to identify the directory as a valid Helm chart:

apiVersion: v2
name: hcp-baremetal-fleet
description: Enterprise Helm Chart for Hosted Control Planes on KubeVirt
version: 1.0.0

Inside the templates folder, we define OpenShift Virtualization and Advanced Cluster Management manifests.

The first manifest is hostedcluster.yaml, which looks similar to the following:

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: {{ .Values.clusterName }}
  namespace: clusters
  labels:
    cluster.open-cluster-management.io/clusterset: {{ .Values.clusterSet | default "default" }}
spec:
  autoscaling:
    scaling: ScaleUpAndScaleDown
  channel: {{ .Values.channel }}
  infraID: {{ .Values.clusterName }}
  infrastructureAvailabilityPolicy: HighlyAvailable
  controllerAvailabilityPolicy: HighlyAvailable
  dns:
    baseDomain: {{ .Values.dns.baseDomain }}
  etcd:
    managed:
      storage:
        type: PersistentVolume
        persistentVolume:
          size: {{ .Values.etcd.size | default "8Gi" }}
          storageClassName: {{ .Values.etcd.storageClass }}
    managementType: Managed
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
    - cidr: {{ .Values.network.clusterCidr }}
    serviceNetwork:
    - cidr: {{ .Values.network.serviceCidr }}
  platform:
    type: KubeVirt
    kubevirt:
      baseDomainPassthrough: true
  pullSecret:
    name: pullsecret-cluster-{{ .Values.clusterName }}
  sshKey:
    name: sshkey-cluster-{{ .Values.clusterName }}
  release:
    image: quay.io/openshift-release-dev/ocp-release:{{ .Values.ocpVersion }}
  configuration:
    proxy:
      httpProxy: {{ .Values.proxy.http | default "" }}
      httpsProxy: {{ .Values.proxy.https | default "" }}
      noProxy: {{ .Values.proxy.noProxy | default "" }}
  services:
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer

Here is an example of the parameterized nodepool.yaml:

apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  name: {{ .Values.clusterName }}-worker-pool
  namespace: clusters
spec:
  arch: amd64
  clusterName: {{ .Values.clusterName }}
  replicas: {{ .Values.workerReplicas | default 3 }}
  management:
    autoRepair: true
    upgradeType: Replace
  release:
    image: quay.io/openshift-release-dev/ocp-release:{{ .Values.ocpVersion }}
  platform:
    type: KubeVirt
    kubevirt:
      compute:
        cores: {{ .Values.compute.cores | default 8 }}
        memory: {{ .Values.compute.memory | default "32Gi" }}
      attachDefaultNetwork: false
      rootVolume:
        type: Persistent
        persistent:
          size: {{ .Values.compute.diskSize | default "120Gi" }}
          accessModes:
            - ReadWriteMany
          storageClass: {{ .Values.compute.storageClass }}
          volumeMode: Block
      additionalNetworks:
        - name: openshift-cnv/{{ .Values.network.vlan }}
  {{- if .Values.nodeLabels }}
  nodeLabels:
{{ toYaml .Values.nodeLabels | indent 4 }}
  {{- end }}
  {{- if .Values.taints }}
  taints:
{{ toYaml .Values.taints | indent 4 }}
  {{- end }}

A critical detail here is attachDefaultNetwork: false combined with additionalNetworks, which will be specified for each NodePool instance (in this example, all the networks have been defined via NMState in the openshift-cnv namespace). This ensures the VMs bypass the default pod network and attach directly to the physical corporate VLAN via Multus CNI.

Note that during a hosted control plane cluster upgrade, worker nodes will be replaced according to the .spec.management.upgradeType: Replace configuration. You can change this value with InPlace if you want to avoid node replacement. See Updating hosted control plane and Updates for node pools for additional guidance.

Finally, we automate the import of the newly provisioned hosted cluster into Red Hat Advanced Cluster Management. The corresponding manifest contains both the ManagedCluster and KlusterletAddonConfig resource definition:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: {{ .Values.clusterName }}
  labels:
    cloud: BareMetal
    vendor: OpenShift
spec:
  hubAcceptsClient: true
---
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
  name: {{ .Values.clusterName }}
  namespace: {{ .Values.clusterName }}
spec:
  clusterName: {{ .Values.clusterName }}
  clusterNamespace: {{ .Values.clusterName }}
  clusterLabels:
    cloud: BareMetal
    vendor: OpenShift
  applicationManager:
    enabled: true
  policyController:
    enabled: true
  searchCollector:
    enabled: true
  certPolicyController:
    enabled: true

Step 4: The clusters

The clusters directory is the only part of the repository that platform operators will interact with. To set up a new cluster, you only need to populate a lightweight YAML file.

Here is the definition for clusters/ocp-dev-01.yaml:

clusterName: ocp-dev-01
ocpVersion: 4.20.22-multi
channel: fast-4.20
dns:
  baseDomain: apps.mydomain.local
etcd:
  storageClass: ocs-storagecluster-ceph-rbd
  size: 8Gi
workerReplicas: 3
compute:
  cores: 12
  memory: 64Gi
  diskSize: 250Gi
  storageClass: ocs-storagecluster-ceph-rbd
network:
  clusterCidr: 10.128.0.0/14
  serviceCidr: 172.30.0.0/16
  vlan: vlan-123
# Optional: uncomment and edit the following section to add node labels and/or taints
#nodeLabels:
#  node-role.kubernetes.io/infra: ""
#taints:
#  - effect: NoSchedule
#    key: infra
#    value: reserved
#  - effect: NoExecute
#    key: infra
#    value: reserved
proxy:
  http: http://proxy.enterprise.lan:8080
  https: http://proxy.enterprise.lan:8080
  noProxy: .mydomain.local,10.28.0.0/14,172.30.0.0/16,192.168.1.0/24,localhost

Important note on proxy configuration

The noProxy list shown here is just an example. When deploying hosted control planes behind a corporate proxy, it is mandatory to bypass the proxy not only for the hosted cluster's internal CIDRs, but also for the management cluster's machine, cluster, and service networks and API server domain. Since the control plane pods run physically on the management cluster, failing to include these subnets will route internal control plane traffic through the corporate proxy, breaking the cluster's core communication.

As soon as this file is committed and pushed to the Git repository, the Argo CD ApplicationSet detects it and automatically provisions the new hosted control plane. Upgrading a cluster, scaling workers, or changing network attachments is as simple as modifying this single file.

Deployment and Day 2 validation

At this point, the synchronization process begins. OpenShift GitOps creates a new Application for each cluster defined in the clusters folder and then applies the corresponding resources on the cluster.

To monitor Argo CD applications and resources, log in to the Argo CD web console. Retrieve the console URL by logging in to the management bare-metal cluster and typing:

oc get route -n openshift-gitops

The Argo CD Application in the web console will look similar to Figure 4.

ArgoCD Application topology view
Figure 4: Argo CD Application status in the web console.

Then, follow these steps to monitor hosted cluster creation from the management cluster.

Step 1: Verify the control plane pods and cluster operators

When creating a new cluster, HyperShift automatically sets up a namespace with the naming convention clusters-<hosted_cluster_name>. To monitor control plane pods, launch the following command:

watch oc get pods -n clusters-ocp-dev-01

To check hosted control plane cluster operators, the kubeconfig for the new cluster can be retrieved from the clusters namespace and used to access the new hosted cluster:

oc extract -n clusters-<hosted_cluster_name> \ secret/<hosted_cluster_name>-admin-kubeconfig \ --to=<path-to-hosted-cluster-kubeconfig> --confirm
watch oc get co –-kubeconfig=<path-to-hosted-cluster-kubeconfig>

Finally, you can also monitor the creation process from the Advanced Cluster Management web console (Fleet Management view of the Red Hat OpenShift web console) of your management cluster, as shown in Figure 5.

Hosted cluster status in Red Hat Advanced Cluster Management dashboard
Figure 5: Advanced Cluster Management web console fleet management view.

You can also expand the Control plane status section to monitor cluster operators, as shown in Figure 6.

Hosted cluster operators status in Red Hat Advanced Cluster Management dashboard
Figure 6: Expanded control plane status showing cluster operators.

Step 2: Monitor worker VM creation and node readiness

Check VM creation by launching the following command on the management cluster:

oc get vmi -n clusters-ocp-dev-01

You should also see an IP address associated with each VirtualMachineInstance (VMI). If VMs do not receive an IP address, check your network and the DHCP configuration.

Command output should be similar to the following:

NAME                           AGE   PHASE     IP            NODENAME                       READY
ocp-dev-01-worker-pool-2rq7m   15m   Running   10.128.2.45   worker-0.mgmt.mydomain.local   True
ocp-dev-01-worker-pool-4l425   15m   Running   10.128.2.46   worker-1.mgmt.mydomain.local   True
ocp-dev-01-worker-pool-4zfd8   15m   Running   10.130.0.23   worker-2.mgmt.mydomain.local   True

If you want to monitor VM status from the Red Hat OpenShift web console of your management cluster, you need to switch to the Administrator view and move to the Virtualization → VirtualMachines section as shown in Figure 7.

Hosted cluster status in Red Hat Advanced Cluster Management dashboard
Figure 7: VirtualMachines view in the OpenShift web console.

Once VMs are provisioned correctly, worker nodes appear in the hosted cluster:

oc get nodes --kubeconfig=<path-to-hosted-cluster-kubeconfig>

You can monitor nodes joining the cluster from the Nodes tab in the Red Hat Advanced Cluster Management dashboard, as shown in Figure 8.

Hosted cluster nodes status in Red Hat Advanced Cluster Management dashboard
Figure 8: Nodes tab in the Red Hat Advanced Cluster Management dashboard.

Step 3: Access the hosted cluster

Wait for the installation to complete. Launch the following command to monitor the progress:

watch "oc get co --kubeconfig=<path-to-hosted-cluster-kubeconfig>; oc get nodes --kubeconfig=<path-to-hosted-cluster-kubeconfig>; oc get clusterversion --kubeconfig=<path-to-hosted-cluster-kubeconfig>"

You can also monitor hosted cluster installation from the Red Hat Advanced Cluster Management web console.

When cluster installation is complete, you can access the hosted cluster using the downloaded kubeconfig.

Conclusion

The shift from traditional, node-heavy control planes to hosted control planes on bare metal is a game-changer for infrastructure efficiency.

Scaling Red Hat OpenShift doesn't have to mean endlessly racking servers and manually configuring networks. By decoupling the control plane with hosted control planes, virtualizing the worker nodes with OpenShift Virtualization, and orchestrating everything through a declarative GitOps model, infrastructure complexity is finally tamed.

With Helm handling enterprise proxies and L2 networks, and an Argo CD ApplicationSet managing the deployment, provisioning a security-focused, production-grade cluster is now as simple as a Git push. It’s an architecture designed for cost-efficiency, built for speed, and structured for security in enterprise environments.