About Argo CD ApplicationSet and SCM Provider generator

The ApplicationSet controller is a part of Argo CD that adds support for an ApplicationSet CustomResourceDefinition (CRD). The ApplicationSet controller helps to add Application automation and improve multicluster support and cluster multitenant support within Argo CD.

ApplicationSet provides the following benefits:

• The ability to use a single Kubernetes manifest to target multiple Kubernetes clusters and deploy multiple applications from one or multiple Git repositories with Argo CD.
• Improved support for monorepos. In the context of Argo CD, a monorepo is multiple Argo CD Application resources defined within a single Git repository.
• Within multitenant clusters, improves the ability of individual cluster tenants to deploy applications using Argo CD (without needing to involve privileged cluster administrators in enabling the destination clusters/namespaces).

You can view all the supported fields for ApplicationSet in the Argo CD documentation.

ApplicationSet generators

Generators generate parameters and are rendered into the template field of the ApplicationSet resource. ApplicationSet supports a variety of generators, as described below:

• List generator: Generates parameters based on an arbitrary list of key/value pairs.
• Cluster generator: Allows you to target Argo CD applications to clusters based on the list of clusters defined within (and managed by) Argo CD.
• Git generator: Allows you to create applications based on files within a Git repository or based on the directory structure of a Git repository.
• Matrix generator: Can be used to combine the generated parameters of two separate generators.
• Merge generator: Can be used to merge the generated parameters of two or more generators.
• SCM Provider generator: Uses the API of a source code management (SCM) provider (e.g., GitHub) to discover repositories within an organization automatically.
• Pull Request generator: Uses the API of a Source-Code-Management-as-a-Service (SCMaaS) provider (e.g., GitHub) to discover open pull requests within a repository automatically.
• Cluster Decision Resource generator: Used to interface with Kubernetes custom resources that use custom resource-specific logic to decide which set of Argo CD clusters to deploy to.
• Plug-in generator: Makes RPC HTTP requests to provide parameters.

This article focuses exclusively on the SCM Provider generator and its use cases.

Source code management

SCM is used to track modifications to a source code repository. SCM tracks a running history of changes to a code base and helps resolve conflicts when merging updates from multiple contributors. SCM is also synonymous with version control.

As software projects grow in lines of code and contributor headcount, the costs of communication overhead and management complexity also grow. SCM is a critical tool to alleviate the organizational strain of growing development costs.

SCM Provider generator

The SCM Provider generator uses the API of a SCMaaS provider such as GitHub to discover repositories within an organization automatically. This fits well with GitOps layout patterns that split microservices across many repositories.

As of Argo CD version 2.8, the supported SCM providers are:

• GitHub
• GitLab 
• Gitea
• Bitbucket 
• Azure DevOps
• Bitbucket Cloud

You can configure the ApplicationSet with SCM Provider as follows:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
spec:
  generators:
  - scmProvider:
      # Which protocol to clone using.
      cloneProtocol: ssh
      # See below for provider specific options.
      github:
        # ...

cloneProtocol: defines the protocol to use for the SCM URL. The default is provider-specific, but ssh if possible.

The provider's parameters are described in the Argo CD documentation. We will use GitHub for the example in this article.

GitHub

The GitHub mode uses the GitHub API to scan an organization in either github.com or GitHub Enterprise:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  generators:
  - scmProvider:
      github:
        organization: argoproj
      cloneProtocol: https
      filters:
      - repositoryMatch: example-apps
  template:
    metadata:
      name: '{{ repository }}-guestbook'
    spec:
      project: "default"
      source:
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
        path: guestbook
      destination:
        server: https://kubernetes.default.svc
        namespace: guestbook

The parameters are explained below:

• Organization: In the example above, we used argoproj. If you have multiple organizations, use multiple generators.
• api: This field is optional; however, if you are using GitHub Enterprise, you will need to specify the URL to access the API. In our example, we are not using GitHub Enterprise.
• filters: SCM Provider supports multiple filters. We are using a repositoryMatch filter matching the repository with name example-apps. You can find more filters in the documentation.

Use case

There is an e-commerce website with multiple services for orders, inventory, payments, etc.; each service is stored in a separate Git repository within the same organization. When we deploy the e-commerce website, we want all the services to be deployed simultaneously for the website to start successfully.

In this case, we can use the SCM Provider generator to generate all the services belonging to the same organization. You can configure the ApplicationSet in the following format:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ecommerce-website
spec:
  generators:
  - scmProvider:
      github:
        organization: ecommerceWebsite
      cloneProtocol: https
  template:
    metadata:
      name: '{{ repository }}'
    spec:
      project: "default"
      source:
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
        path: app
      destination:
        server: https://kubernetes.default.svc
        namespace: ecomm

The ApplicationSet above will read all the repositories in the GitHub organization ecommerceWebsite and deploy all the services in the repositories within the path app in each repository.

Pros

Deploying multiple applications and services in the same organization is easy and quick. You can use the SCM Provider generator to allow you to dynamically generate ephemeral environments for testing or prod-like environments easily.

Cons

Once the entire organization is added to the SCM Provider generator, it will pick all the repositories matching the added filtering conditions. If you want to create a new repository within the same organization and do not want it to be deployed, you would need to ensure that the repository does not match any of the filtering conditions added to the SCM Provider generator. This becomes an additional effort if the SCM generator is not configured properly.

Conclusion

The ApplicationSet custom resource helps you easily and quickly manage multitenant or cluster application deployment. The SCM Provider generator is helpful when you want to manage all repositories of a known pattern within an organization and deploy resources to Argo CD by automatically identifying changes to any of the repositories in the organization.