In the dynamic world of cloud-native development, managing a single Kubernetes cluster can be challenging. But when you scale up to dozens, hundreds, or even thousands of clusters across various environments and cloud providers, complexity skyrockets. This is where Red Hat Advanced Cluster Management for Kubernetes can help, providing a centralized platform to configure all your clusters.
Advanced Cluster Management for Kubernetes offers a comprehensive solution for managing the entire lifecycle of your Kubernetes clusters, from creation and import to upgrades and decommissioning. It provides a unified control plane to monitor cluster health, enforce policies, and manage configurations consistently across your distributed fleet.
But what if you need to apply slightly different configurations or deploy different application versions based on specific characteristics of a cluster? This is where the powerful combination of Advanced Cluster Management for Kubernetes's cluster labeling with ApplicationSets and Helm comes into play.
Managing clusters with Advanced Cluster Management for Kubernetes
Advanced Cluster Management for Kubernetes offers a single point of control for any CNCF-conformant Kubernetes cluster. This allows you to seamlessly manage your entire fleet, whether your clusters are running in a datacenter, a public cloud, or at the edge. You can easily:
- Create and import clusters: Provision new OpenShift clusters or import. existing Kubernetes clusters with a few clicks or through Infrastructure as Code.
- Monitor and observe: Gain visibility into the health, capacity, and performance of all your clusters from a central dashboard.
- Govern and strengthen: Enforce consistent security policies, compliance standards, and configurations across your entire fleet, reducing risk and ensuring adherence to organizational guidelines.
- Automate operations: Streamline routine tasks and manage the lifecycle of your clusters programmatically, embracing GitOps principles.
Beyond basic deployments with ApplicationSets and Helm
While Advanced Cluster Management for Kubernetes provides robust cluster management, deploying and managing applications across a heterogeneous environment requires precision control. This is where Red Hat OpenShift GitOps, powered by Argo CD, and its ApplicationSet feature become indispensable.
An ApplicationSet is a powerful extension to Argo CD that enables you to automate the generation of Argo CD applications. Instead of manually creating an Argo CD application for each cluster or each variation of your deployment, an ApplicationSet allows you to define a template that dynamically generates these applications based on various "generators" (one of the most powerful being the clusterDecisionResource generator).
Helm, the package manager for Kubernetes, enhances this capability. A Helm chart provides a templating mechanism for a Kubernetes manifest, allowing you to define configurable and reusable application packages. All of this is even more powerful when you combine Helm with ApplicationSets, and leverage cluster labels.
Advanced Cluster Management for Kubernetes with Red Hat OpenShift GitOps
This section covers how to set up Advanced Cluster Management for Kubernetes and Red Hat OpenShift GitOps integration.
Prerequisites
Install Red Hat OpenShift GitOps Operator in the cluster. You can find more in the documentation.
Configuration
First, create the ManagedClusterSet:
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
name: managed-clusters
spec:
clusterSelector:
selectorType: ExclusiveClusterSetLabel
Next, create the ManagedClusterSetBinding:
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
name: managed-clusters
namespace: openshift-gitops
spec:
clusterSet: managed-clusters
Create the Placement:
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
name: gitops
namespace: openshift-gitops
spec:
clusterSets:
- "managed-clusters"
Add the desired cluster into the ClusterSet:
- Click Infrastructure > Clusters > ClusterSets tab > managed-clusters
Click the Cluster List tab and then select the desired cluster to manage with GitOps. Click the Review button (Figure 1).

Figure 1: The two clusters, development and local-cluster, running on AWS. - Review the changes, and then click Save.
Finally, create the GitOpsCluster:
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
name: argo-acm-clusters
namespace: openshift-gitops
spec:
argoServer:
cluster: local-cluster
argoNamespace: openshift-gitops
placementRef:
kind: Placement
apiVersion: cluster.open-cluster-management.io/v1beta1
name: gitops
namespace: openshift-gitopsConditional deployments with cluster labels
Imagine you have a common application that needs to be deployed across your development, staging, and production clusters. Each environment might require different resource limits, logging configurations, or even a different image tag. Manually maintaining separate Helm charts or values files for each scenario quickly becomes unwieldy and is prone to error.
If you have multiple clusters, you can create distinct namespaces for each (and this could be applied to any scenario by creating other Helm charts). You can find all the configuration files used in this blog post in this GitHub repository.
Here's where Advanced Cluster Management for Kubernetes's cluster labels provide the intelligence for your GitOps deployments:
Advanced Cluster Management for Kubernetes allows you to apply arbitrary labels to your managed clusters. These labels act as metadata, classifying your clusters based on characteristics like environment: dev, environment: prod, region: eu-west-1, owner: team-x, or purpose: logging.
For example, you might label your development cluster as environment: dev and your production cluster as environment: prod.
oc label managedcluster <cluster-name> environment=devYou can do this step through the Advanced Cluster Management for Kubernetes web interface. First, select the cluster, and then select Edit labels and add the required label.
Using a template (applicationset.yaml in the Git repository), this application is created:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cluster-config
namespace: openshift-gitops
spec:
generators:
- matrix:
generators:
- merge:
generators:
- clusterDecisionResource:
configMapRef: acm-placement
labelSelector:
matchLabels:
cluster.open-cluster-management.io/placement: gitops
- clusters: {}
mergeKeys:
- server
- list:
elements:
- template: app-of-apps
template:
metadata:
name: "cluster-{{clusterName}}-config"
labels:
environment: "{{metadata.labels.environment}}"
spec:
project: default
source:
repoURL: https://github.com/misanche/acm-gitops.git
targetRevision: "main"
path: base/components/{{template}}
helm:
ignoreMissingValueFiles: true
valueFiles:
- /conf/{{metadata.labels.environment}}/conf.yaml
destination:
server: "{{server}}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- Validate=trueThis is a sophisticated use of ApplicationSet generators, combining matrix, merge, clusterDecisionResource, clusters, and list to achieve flexible, dynamic deployments. The core idea of ApplicationSet generators is to produce a list of parameters (key-value pairs) used to populate the template section, effectively creating multiple Argo CD Application resources.
Overview of generators
matrixgenerator: This generator takes two or more other generators and creates a Cartesian product of their outputs. If Generator A produces {A1, A2} and Generator B produces {B1, B2}, then the matrix produces {A1+B1, A1+B2, A2+B1, A2+B2}.mergegenerator: This generator also takes two or more other generators. It merges the outputs of its child generators based on specifiedmergeKeys. For each set of parameters where themergeKeysmatch across the child generators, it combines their parameters into a single output set.clusterDecisionResourcegenerator: This generator integrates with Advanced Cluster Management for Kubernetes, and discovers clusters targeted by a specific Advanced Cluster Management for KubernetesPlacementresource. It outputs parameters related to these clusters, such asname(the cluster name),server(the cluster API server URL), andmetadata.labelsandmetadata.annotations(the labels and annotations applied to theManagedClusterresource in Advanced Cluster Management for Kubernetes).clustersgenerator: This is a simple generator that discovers all clusters known to Argo CD itself. It outputsnameandserverfor each registered cluster.listgenerator: This generator allows you to explicitly define a static list of parameters (elements) to be used by theApplicationSet.
Detailed breakdown of the generators section
From the application defined above, here's the generators section, with comments in the YAML explaining what each item does.
generators:
- matrix: # Top-level generator, combines output of its two child generators
generators:
- merge: # First child of matrix, merges outputs from clusterDecisionResource and clusters
generators:
- clusterDecisionResource: # First child of merge, gets clusters from ACM Placement
configMapRef: acm-placement
labelSelector:
matchLabels:
cluster.open-cluster-management.io/placement: gitops
- clusters: {} # Second child of merge, gets all clusters known to Argo CD
mergeKeys:
- server # Merge key, only merges if 'server' matches between outputs of clusterDecisionResource and clusters
- list: # Second child of matrix, provides a static list of 'template' values
elements:
- template: app-of-apps
Inner merge generator
The merge generator has two sub-generators:
clusterDecisionResource: This looks for clusters targeted by a Advanced Cluster Management for KubernetesPlacementnamedgitops.It does this implicitly, becauseconfigMapRef: acm-placementwould point to a ConfigMap defining the GVK ofPlacement, andlabelSelectoris used to pick the specific Placement instance, often by a label on the Placement itself or a specific name in the ConfigMap. For each cluster selected by this placement, it outputs parameters likename,server, and importantly,metadata.labels(the labels from the Advanced Cluster Management for KubernetesManagedClusterresource).clusters: This generator simply lists all clusters that are currently registered in Argo CD. It outputs theirnameandserver.
The mergeKeys: ["server"] instruction is crucial here. It means that the output of the clusterDecisionResource generator and the clusters generator is only merged if they have the same server value.
mergeKeys: ["server"]
This is a common pattern when integrating Advanced Cluster Management for Kubernetes with Argo CD. Advanced Cluster Management for Kubernetes knows about your managed clusters and their labels. Argo CD knows about your registered clusters (which might include clusters not managed by Advanced Cluster Management for Kubernetes directly). By merging on server, you ensure that:
- Only clusters that are both managed by Advanced Cluster Management for Kubernetes (and targeted by your placement) and registered with Argo CD are considered
- The combined output includes all rich metadata from the Advanced Cluster Management for Kubernetes
ManagedCluster(likemetadata.labels.environment) along with the standardnameandserverthat Argo CD uses.
Output of this merge generator
For each cluster that satisfies both conditions (Advanced Cluster Management for Kubernetes-managed or placed AND registered by Argo CD), this merge generator produces a set of parameters. For example:
- name: my-dev-cluster
server: https://api.dev.example.com:6443
metadata: # (from clusterDecisionResource)
labels:
environment: dev
...
- name: my-prod-cluster
server: https://api.prod.example.com:6443
metadata: # (from clusterDecisionResource)
labels:
environment: prod
...list generator
The list generator at the end of the example YAML file produces a single element with one parameter (template: app-of-app), which is the list generator's output:
- template: app-of-appsOuter matrix generator
The matrix generator takes the output of the merge generator and the list generator, and combines them in a Cartesian product.
Because the list generator only produces one element (template: app-of-apps), the matrix effectively means: For each cluster identified by the merge generator, combine it with template: app-of-apps.
Final output of the generators section
For each cluster discovered by the merge generator (for example, my-dev-cluster, my-prod-cluster), the matrix generator produces a combined set of parameters. For example, the output of my-dev-cluster:
- name: my-dev-cluster
server: https://api.dev.example.com:6443
metadata:
labels:
environment: dev
template: app-of-appsFor my-prod-cluster:
- name: my-prod-cluster
server: https://api.prod.example.com:6443
metadata:
labels:
environment: prod
template: app-of-appsHow these parameters are used in the template section:
The template section of the ApplicationSet uses these generated parameters to create individual Argo CD Application resources.
metadata.name: "cluster-{{clusterName}}-config": This becomescluster-my-dev-cluster-configandcluster-my-prod-cluster-config.metadata.labels: environment: "{{metadata.labels.environment}}": This pulls theenvironmentlabel directly from themetadata.labelsparameter provided by theclusterDecisionResourcegenerator (from the Advanced Cluster Management for KubernetesManagedCluster's labels). So the generated Application formy-dev-clusterhas the labelenvironment: dev.path: base/components/{{template}}: This resolves tobase/components/app-of-appsfor all generated applications, directing them to a specific path within your Git repository.destination.server: "{{server}}": This uses theserverparameter (the API server URL) to correctly point the Argo CD Application to the target cluster.valueFiles: - /conf/{{metadata.labels.environment}}/conf.yaml: This is the most powerful part for this use case! It dynamically constructs the path to the HelmvalueFilesbased on theenvironmentlabel of the target cluster.- For
my-dev-cluster(withenvironment: dev), it uses/conf/dev/conf.yaml - For
my-prod-cluster(withenvironment: prod), it uses/conf/prod/conf.yaml
- For
In summary, this ApplicationSet design allows you to:
- Dynamically discover clusters that are managed by Advanced Cluster Management for Kubernetes (and targeted by a specific placement) and registered with Argo CD.
- Access cluster-specific metadata, particularly labels from Advanced Cluster Management for Kubernetes.
- Generate multiple Argo CD applications for these clusters.
- Apply different Helm configurations (with
valueFiles) based on theenvironmentlabel of each target cluster, ensuring environment-specific deployments from a single source chart. - Use a specific component path (
app-of-apps) for all these generated applications. Theapp-of-appshelm charts uses the Argo CD app of apps pattern to generate dynamic Argo CD applications.
If you access Argo CD you see the following output with the application generated for the managed cluster development (Figure 2).

Conclusion
By combining the powerful cluster management capabilities of Red Hat Advanced Cluster Management with the GitOps automation of Red Hat OpenShift GitOps (Argo CD) and its ApplicationSets, you can achieve highly intelligent and scalable application deployments. Leveraging cluster labels to dynamically select Helm values or apply conditional logic within your charts empowers you to maintain a single source of truth in Git while catering to the unique requirements of your diverse Kubernetes environments. This approach streamlines operations, reduces manual errors, and accelerates your path to delivering applications efficiently across your entire hybrid cloud landscape.
Resources: