Smart deployments at scale: Leveraging ApplicationSets and Helm with cluster labels in Red Hat Advanced Cluster Management for Kubernetes

In the dynamic world of cloud-native development, managing a single Kubernetes cluster can be challenging. But when you scale up to dozens, hundreds, or even thousands of clusters across various environments and cloud providers, complexity skyrockets. This is where Red Hat Advanced Cluster Management for Kubernetes can help, providing a centralized platform to configure all your clusters.

Advanced Cluster Management for Kubernetes offers a comprehensive solution for managing the entire lifecycle of your Kubernetes clusters, from creation and import to upgrades and decommissioning. It provides a unified control plane to monitor cluster health, enforce policies, and manage configurations consistently across your distributed fleet.

But what if you need to apply slightly different configurations or deploy different application versions based on specific characteristics of a cluster? This is where the powerful combination of Advanced Cluster Management for Kubernetes's cluster labeling with ApplicationSets and Helm comes into play.

Managing clusters with Advanced Cluster Management for Kubernetes

Advanced Cluster Management for Kubernetes offers a single point of control for any CNCF-conformant Kubernetes cluster. This allows you to seamlessly manage your entire fleet, whether your clusters are running in a datacenter, a public cloud, or at the edge. You can easily:

• Create and import clusters: Provision new OpenShift clusters or import. existing Kubernetes clusters with a few clicks or through Infrastructure as Code.
• Monitor and observe: Gain visibility into the health, capacity, and performance of all your clusters from a central dashboard.
• Govern and strengthen: Enforce consistent security policies, compliance standards, and configurations across your entire fleet, reducing risk and ensuring adherence to organizational guidelines.
• Automate operations: Streamline routine tasks and manage the lifecycle of your clusters programmatically, embracing GitOps principles.

Beyond basic deployments with ApplicationSets and Helm

While Advanced Cluster Management for Kubernetes provides robust cluster management, deploying and managing applications across a heterogeneous environment requires precision control. This is where Red Hat OpenShift GitOps, powered by Argo CD, and its ApplicationSet feature become indispensable.

An ApplicationSet is a powerful extension to Argo CD that enables you to automate the generation of Argo CD applications. Instead of manually creating an Argo CD application for each cluster or each variation of your deployment, an ApplicationSet allows you to define a template that dynamically generates these applications based on various "generators" (one of the most powerful being the clusterDecisionResource generator).

Helm, the package manager for Kubernetes, enhances this capability. A Helm chart provides a templating mechanism for a Kubernetes manifest, allowing you to define configurable and reusable application packages. All of this is even more powerful when you combine Helm with ApplicationSets, and leverage cluster labels.

Advanced Cluster Management for Kubernetes with Red Hat OpenShift GitOps

This section covers how to set up Advanced Cluster Management for Kubernetes and Red Hat OpenShift GitOps integration.

Prerequisites

Install Red Hat OpenShift GitOps Operator in the cluster. You can find more in the documentation.

Configuration

First, create the ManagedClusterSet:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: managed-clusters
spec:
  clusterSelector:
    selectorType: ExclusiveClusterSetLabel

Next, create the ManagedClusterSetBinding:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: managed-clusters
  namespace: openshift-gitops
spec:
  clusterSet: managed-clusters

Create the Placement:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: gitops
  namespace: openshift-gitops
spec:
  clusterSets:
    - "managed-clusters"

Add the desired cluster into the ClusterSet:

1. Click Infrastructure > Clusters > ClusterSets tab > managed-clusters

2. Click the Cluster List tab and then select the desired cluster to manage with GitOps. Click the Review button (Figure 1). 

   

3. Review the changes, and then click Save.

Finally, create the GitOpsCluster:

apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: argo-acm-clusters
  namespace: openshift-gitops
spec:
  argoServer:
    cluster: local-cluster
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: gitops
    namespace: openshift-gitops

Conditional deployments with cluster labels

Imagine you have a common application that needs to be deployed across your development, staging, and production clusters. Each environment might require different resource limits, logging configurations, or even a different image tag. Manually maintaining separate Helm charts or values files for each scenario quickly becomes unwieldy and is prone to error.

If you have multiple clusters, you can create distinct namespaces for each (and this could be applied to any scenario by creating other Helm charts). You can find all the configuration files used in this blog post in this GitHub repository.

Here's where Advanced Cluster Management for Kubernetes's cluster labels provide the intelligence for your GitOps deployments:

Advanced Cluster Management for Kubernetes allows you to apply arbitrary labels to your managed clusters. These labels act as metadata, classifying your clusters based on characteristics like environment: dev, environment: prod, region: eu-west-1, owner: team-x, or purpose: logging.
For example, you might label your development cluster as environment: dev and your production cluster as environment: prod.

oc label managedcluster <cluster-name> environment=dev

You can do this step through the Advanced Cluster Management for Kubernetes web interface. First, select the cluster, and then select Edit labels and add the required label.

Using a template (applicationset.yaml in the Git repository), this application is created:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-config
  namespace: openshift-gitops
spec:
  generators:
  - matrix:
      generators:
      - merge:
          generators:
          - clusterDecisionResource:
              configMapRef: acm-placement
              labelSelector:
                matchLabels:
                  cluster.open-cluster-management.io/placement: gitops
          - clusters: {}
          mergeKeys:
          - server
      - list:
          elements:
            - template: app-of-apps
  template:
    metadata:
      name: "cluster-{{clusterName}}-config"
      labels:
        environment: "{{metadata.labels.environment}}" 
    spec:
      project: default
      source:
        repoURL: https://github.com/misanche/acm-gitops.git
        targetRevision: "main"
        path: base/components/{{template}}
        helm:
          ignoreMissingValueFiles: true
          valueFiles:
            - /conf/{{metadata.labels.environment}}/conf.yaml
      destination:
        server: "{{server}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - Validate=true

This is a sophisticated use of ApplicationSet generators, combining matrix, merge, clusterDecisionResource, clusters, and list to achieve flexible, dynamic deployments. The core idea of ApplicationSet generators is to produce a list of parameters (key-value pairs) used to populate the template section, effectively creating multiple Argo CD Application resources.

Overview of generators

• matrix generator: This generator takes two or more other generators and creates a Cartesian product of their outputs. If Generator A produces {A1, A2} and Generator B produces {B1, B2}, then the matrix produces {A1+B1, A1+B2, A2+B1, A2+B2}.
• merge generator: This generator also takes two or more other generators. It merges the outputs of its child generators based on specified mergeKeys. For each set of parameters where the mergeKeys match across the child generators, it combines their parameters into a single output set.
• clusterDecisionResource generator: This generator integrates with Advanced Cluster Management for Kubernetes, and discovers clusters targeted by a specific Advanced Cluster Management for Kubernetes Placement resource. It outputs parameters related to these clusters, such as name (the cluster name), server (the cluster API server URL), and metadata.labels and metadata.annotations (the labels and annotations applied to the ManagedCluster resource in Advanced Cluster Management for Kubernetes).
• clusters generator: This is a simple generator that discovers all clusters known to Argo CD itself. It outputs name and server for each registered cluster.
• list generator: This generator allows you to explicitly define a static list of parameters (elements) to be used by the ApplicationSet.

Detailed breakdown of the generators section

From the application defined above, here's the generators section, with comments in the YAML explaining what each item does.

  generators:
  - matrix: # Top-level generator, combines output of its two child generators
      generators:
      - merge: # First child of matrix, merges outputs from clusterDecisionResource and clusters
          generators:
          - clusterDecisionResource: # First child of merge, gets clusters from ACM Placement
              configMapRef: acm-placement
              labelSelector:
                matchLabels:
                  cluster.open-cluster-management.io/placement: gitops
          - clusters: {} # Second child of merge, gets all clusters known to Argo CD
          mergeKeys:
          - server # Merge key, only merges if 'server' matches between outputs of clusterDecisionResource and clusters
      - list: # Second child of matrix, provides a static list of 'template' values
          elements:
            - template: app-of-apps

Inner merge generator

The merge generator has two sub-generators:

• clusterDecisionResource: This looks for clusters targeted by a Advanced Cluster Management for Kubernetes Placement named gitops. It does this implicitly, because configMapRef: acm-placement would point to a ConfigMap defining the GVK of Placement, and labelSelector is used to pick the specific Placement instance, often by a label on the Placement itself or a specific name in the ConfigMap. For each cluster selected by this placement, it outputs parameters like name, server, and importantly, metadata.labels (the labels from the Advanced Cluster Management for Kubernetes ManagedCluster resource).
• clusters: This generator simply lists all clusters that are currently registered in Argo CD. It outputs their name and server.

The mergeKeys: ["server"] instruction is crucial here. It means that the output of the clusterDecisionResource generator and the clusters generator is only merged if they have the same server value.

mergeKeys: ["server"]

This is a common pattern when integrating Advanced Cluster Management for Kubernetes with Argo CD. Advanced Cluster Management for Kubernetes knows about your managed clusters and their labels. Argo CD knows about your registered clusters (which might include clusters not managed by Advanced Cluster Management for Kubernetes directly). By merging on server, you ensure that:

• Only clusters that are both managed by Advanced Cluster Management for Kubernetes (and targeted by your placement) and registered with Argo CD are considered
• The combined output includes all rich metadata from the Advanced Cluster Management for Kubernetes ManagedCluster (like metadata.labels.environment) along with the standard name and server that Argo CD uses.

Output of this merge generator

For each cluster that satisfies both conditions (Advanced Cluster Management for Kubernetes-managed or placed AND registered by Argo CD), this merge generator produces a set of parameters. For example:

- name: my-dev-cluster
  server: https://api.dev.example.com:6443
  metadata: # (from clusterDecisionResource)
    labels:
      environment: dev
      ...
- name: my-prod-cluster
  server: https://api.prod.example.com:6443
  metadata: # (from clusterDecisionResource)
    labels:
      environment: prod
      ...

list generator

The list generator at the end of the example YAML file produces a single element with one parameter (template: app-of-app), which is the list generator's output:

- template: app-of-apps

Outer matrix generator

The matrix generator takes the output of the merge generator and the list generator, and combines them in a Cartesian product.

Because the list generator only produces one element (template: app-of-apps), the matrix effectively means: For each cluster identified by the merge generator, combine it with template: app-of-apps.

Final output of the generators section

For each cluster discovered by the merge generator (for example, my-dev-cluster, my-prod-cluster), the matrix generator produces a combined set of parameters. For example, the output of my-dev-cluster:

- name: my-dev-cluster
  server: https://api.dev.example.com:6443
  metadata:
    labels:
      environment: dev
  template: app-of-apps

For my-prod-cluster:

- name: my-prod-cluster
  server: https://api.prod.example.com:6443
  metadata:
    labels:
      environment: prod
  template: app-of-apps

How these parameters are used in the template section:

The template section of the ApplicationSet uses these generated parameters to create individual Argo CD Application resources.

• metadata.name: "cluster-{{clusterName}}-config": This becomes cluster-my-dev-cluster-config and cluster-my-prod-cluster-config.
• metadata.labels: environment: "{{metadata.labels.environment}}": This pulls the environment label directly from the metadata.labels parameter provided by the clusterDecisionResource generator (from the Advanced Cluster Management for Kubernetes ManagedCluster's labels). So the generated Application for my-dev-cluster has the label environment: dev.
• path: base/components/{{template}}: This resolves to base/components/app-of-apps for all generated applications, directing them to a specific path within your Git repository.
• destination.server: "{{server}}": This uses the server parameter (the API server URL) to correctly point the Argo CD Application to the target cluster.
• valueFiles: - /conf/{{metadata.labels.environment}}/conf.yaml: This is the most powerful part for this use case! It dynamically constructs the path to the Helm valueFiles based on the environment label of the target cluster.
  • For my-dev-cluster (with environment: dev), it uses /conf/dev/conf.yaml
  • For my-prod-cluster (with environment: prod), it uses /conf/prod/conf.yaml

In summary, this ApplicationSet design allows you to:

• Dynamically discover clusters that are managed by Advanced Cluster Management for Kubernetes (and targeted by a specific placement) and registered with Argo CD.
• Access cluster-specific metadata, particularly labels from Advanced Cluster Management for Kubernetes.
• Generate multiple Argo CD applications for these clusters.
• Apply different Helm configurations (with valueFiles) based on the environment label of each target cluster, ensuring environment-specific deployments from a single source chart.
• Use a specific component path (app-of-apps) for all these generated applications. The app-of-apps helm charts uses the Argo CD app of apps pattern to generate dynamic Argo CD applications.

If you access Argo CD you see the following output with the application generated for the managed cluster development (Figure 2).

Conclusion

By combining the powerful cluster management capabilities of Red Hat Advanced Cluster Management with the GitOps automation of Red Hat OpenShift GitOps (Argo CD) and its ApplicationSets, you can achieve highly intelligent and scalable application deployments. Leveraging cluster labels to dynamically select Helm values or apply conditional logic within your charts empowers you to maintain a single source of truth in Git while catering to the unique requirements of your diverse Kubernetes environments. This approach streamlines operations, reduces manual errors, and accelerates your path to delivering applications efficiently across your entire hybrid cloud landscape.

Resources:

• Argo CD generators
• Red Hat Advanced Cluster Management GitOps