The Red Hat OpenShift GitOps operator 1.20.2 brings a whole new way of managing TLS certificates for Argo CD to trust. It builds upon the existing mechanisms and complements them nicely to make managing the trusted hosts, certificates, or certificate authorities (CAs) easier.
Argo CD source trust management
When pulling your application sources over TLS, Argo CD always verifies the host certificate to make sure it communicates with a trusted party. The Argo CD repository server is the component responsible for fetching application sources (i.e., from Git) and building the resulting Kubernetes resource manifests (by Kustomize, Helm or alike). It manages the TLS trust of the source repositories and registries using two distinct mechanisms: trusting the Mozilla CA bundle shipped inside the Argo CD images and allowing administrators to pin any additional certificates they choose to trust.
First, let’s have a detailed look at how they work in concert.
CAs from Mozilla bundle
The containers running the repository server populate with the CA bundle maintained by Mozilla. This automatically makes a large part of the internet trusted through the established CAs in the same way your web browser or operating system trusts them. The popular source code hosting services, Helm, or OCI registries are trusted this way.
When installing the bundle in the repository server and its config management plug-ins in a traditional way, the full set of the certificates is available to tools like Git or Helm when fetching and generating your Kubernetes resource manifests.
Certificate pinning through config map
Sometimes it is necessary to trust TLS hosts that Mozilla does not like when the infrastructure in an organization is internal, and its certificates likely are not signed by any of the common certificate authorities.
To support this use-case, Argo CD exposes a config map named argocd-tls-certs-cm, enabling users to map hostnames to PEM-encoded TLS certificate data. It uses such a certificate every time when fetching an application source from a given host.
The communication with hosts that are not pinned, will default to using the Mozilla CA bundle. So you can combine both approaches, allowing users to use a mixture of repositories, either publicly trusted or pinned explicitly. The important distinction is that, unlike the pinned certificates, the certificates in the bundle are not restricted to specific hostnames by Argo CD.
Use cases
We went into a somewhat detailed explanation of the inner workings, so we can see where these two approaches lead to unexpected challenges. Consider following real-world use cases.
I would like to trust an internal certificate authority or a wildcard certificate.
There is a great reason to use such certificates; they automatically trust a number of hosts by using a single PEM entry, simplifying certificate management significantly. Furthermore, they cryptographically ensure the TLS trust for hostnames that may not even exist when adding the certificate to Argo CD.
This is a challenge when pinning such certificates using a particular hostname (as requested in argocd-tls-certs-cm) because this entirely misses the benefit of such certificates. You must pin them multiple times for every host to use. If there are more hosts emerging in the future—say, your registry load balancer starts delegating to a new set of worker hosts—you must add more pinned entries even though their (unchanged) certificate was already trusted for its other hostnames.
I would like to permit my manifest generation process to reach other TLS hosts.
This one sounds quite convoluted, but it is not. There are specific situations when you need such indirect TLS connections such as:
- Consider a Helm chart hosted on a Git server that declares dependencies on other Helm charts hosted on a different server. You can explicitly trust the Git server using certificate pinning, but not the Helm server hosting the dependencies. I mean, you can pin its certificate as well, but Argo CD will not use it. This is because Argo CD passes along the certificate for the host to the location of the application source (the Git server in this case). But it has no knowledge that other TLS connections are made (to fetch the dependencies), nor their hostnames.
- Something very similar can happen with a Kustomize application by fetching external files over HTTPS or by incorporating helmCharts from hosts not implicitly trusted by the Mozilla CA bundle. Argo CD has no idea about such indirect dependencies and does not apply pinned certificates for their hostnames.
- Another analogous situation is the config management plug-ins can make their own TLS connections (again to hosts unknown to Argo CD).
I would like to distrust some certificate authorities trusted by the Mozilla CA bundle.
Whether you require distrusting the entire bundle and rely on pinning exclusively, or you simply want to distrust a handful of public certificate authorities, you will have a hard time doing so with Argo CD alone.
This is necessary to enforce that the GitOps deployment is only using sources from TLS hosts signed by a given certificate authority. It can also come in handy when a public CA loses its trust (e.g., after a key compromise), and you would like to act before the delivery of an updated bundle through an Argo CD upgrade.
Inject certificates to the bundle
With the latest OpenShift GitOps operator, there is a configuration option that permits declaring new certificates directly mixed into the CA certificate bundle for the repository server and its config management plug-ins. That results in merging the custom certificates with the Mozilla provided ones. Thanks to the deployment to the containers, it addresses the challenges we identified earlier. Since the resulting bundle is where different repository server tools like Git and Kustomize expect it, it is used for every TLS connection they make—direct or indirect.
Note that this comes with a downside. The Argo CD administrator cannot explicitly restrict the CA and wildcard certificates in the bundle to individual hostnames. Unlike the pinned ones, injected certificates apply to any host the certificate matches.
Exemplified usage
To modify the CA certificate bundle, configure the operator with the systemCATrust specification. It instructs the operator to read the PEM-encoded certificates from Secrets or ConfigMaps you choose.
apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
name: example-argocd
spec:
repo:
systemCATrust:
secrets:
- name: my-local-cert-secret
items:
- key: key-name-in-the-secret-object
# Must end with .crt
path: desired-file-name-of-the-certificate.crt
configMaps:
- name: my-local-cert-cm
# Key names in the ConfigMap must end with .crt
items: {}
optional: trueYou can specify multiple Secrets, ConfigMaps, or their combination at once. The resulting bundle will combine all certificates plus the original content of the bundle. Alternatively, users can choose to remove all pre-existing certificates from the bundle by .repo.systemCATrust.dropImageCertificates: true. That way, the bundle will contain only the certificates specified here.
Conveniently, it will use all certificates in the Secret or ConfigMap by default, unless the administrator chooses to list them explicitly as items. Each item then refers to a key name in its respective referenced object and needs to specify a path. The path name can have any value. The operator will inject all the items as long as all paths are unique within the systemCATrust declaration and end with the .crt suffix.
Each referenced Secret or ConfigMap, unless declared optional, must exist in the same project where the Argo CD instance runs. The operator will inject all certificates specified in the Secret’s data/stringData values:
apiVersion: v1
kind: Secret
metadata:
name: my-local-cert-secret
type: Opaque
stringData:
key-name-in-the-secret-object: |
-----BEGIN CERTIFICATE-----
... your certificate content ...
-----END CERTIFICATE----... or from the ConfigMap’s data values respectively:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-local-cert-cm
data:
first-ca.crt: |
-----BEGIN CERTIFICATE-----
... your first certificate content ...
-----END CERTIFICATE-----
another-internal-ca.crt: |
-----BEGIN CERTIFICATE-----
... your second certificate content ...
-----END CERTIFICATE-----```To inject the certificates into config management plug-in containers as well, they use the exact same image and version as the repository server. The best way to achieve this is by not specifying the image field for the repository server and its sidecar containers in the ArgoCD custom resource.
To pin, or not to pin?
Despite limited motivation, you can use the CA bundle management offered by OpenShift GitOps and Argo CD certificate pinning together. Users who simply desire to trust a certificate can inject it in the bundle and celebrate the extra benefits they offer.
One new and exciting thing this combination offers is emptying the bundle using dropImageCertificates: true and pinning selected certificates explicitly in argocd-tls-certs-cm. That way, you're permitting only the pinned hostnames for application source URLs, allowing no other access, direct or indirect, even when trusted by a Mozilla CA bundle. This gives the Argo CD administrator the ultimate control of the origin of the application sources that the application authors cannot override.
Final thoughts
Using native Argo CD certificate pinning and OpenShift GitOps operator’s system CA trust differs in several important aspects. Certificate pinning is a native Argo CD functionality that allows users to trust the certificates configured per explicit hostname. The CA bundle injection provided by the operator makes the certificates freely available for TLS connections made to any host. Now you are all set to balance the strictness of certificate pinning with the convenience of the bundle injection to precisely match your desired security posture.
