The external secrets operator (ESO) for Red Hat OpenShift is now generally available. The operator is based on the upstream external-secrets project and is included with Red Hat OpenShift Container Platform and Red Hat OpenShift Platform Plus. The general availability release is available as a day-2 operator for OpenShift 4.20+, installable through the Red Hat Operator Catalog.
Secrets management
For Red Hat, the external secrets operator completes a trio of significant investments in secrets management for OpenShift clusters. Cert-manager became generally available in OpenShift 4.16. The secrets store csi operator followed suit two releases later in 4.18. With the general availability of the external secrets operator, we can now offer a complete suite of products designed to manage secrets in OpenShift. If you are interested in learning more about how Red Hat views secrets management in OpenShift, check out this learning path.
The following is a brief overview of Red Hat’s secrets management portfolio:
- cert-manager: It automatically manages the lifecycle of x.509 certificates within OpenShift. Administrators can supply a PKI of their choice in order to secure network communications in their cluster with managed TLS certificates. We have also added a deep integration with istio based service meshes, including OpenShift Service Mesh using the istio-csr feature. Learn more...
- secrets store csi driver operator (SSCSI): This operator automatically manages application credentials in OpenShift by mounting requested secrets as files within in-memory volumes attached to a running workload. This means that all secrets are ephemeral, and the underlying Kubernetes cluster is completely unaware of the secret. SSCSI is one of the more secure options in our secrets management portfolio as it enables secrets to be consumed by applications without ever storing them in etcd.
- external secrets operator: The newest addition to the suite of secrets management products, the external secrets operator automatically manages Kubernetes secrets containing application credentials within the OpenShift cluster. This brings added security to the standard Kubernetes secret by adding improved event monitoring and permission abstraction.
Improved security
So what does the external secrets operator bring to the table? Kubernetes secrets are just fine, right? Interestingly, the external secrets operator is actually an extension of the same security principles espoused by Kubernetes secrets. Secrets were designed as an abstraction of confidential information away from the application code. Previously, developers needed to embed application credentials into application code or bake credentials into the environment of a container image. But Kubernetes secrets enabled them to simply request these credentials by referencing a secret name within the application’s deployment YAML file. The result is improved security. Credentials are no longer hardcoded into an application and secrets can be restricted (somewhat) from prying eyes using RBAC policies.
The external secrets operator takes this principle one step further. It completely abstracts away the task of managing these secrets in production from application developers. The operator enables DevSecOps teams to manage credentials from a single source of truth like CyberArk Conjur, HashiCorp Vault, or any of the secrets management tools provided by the major Cloud Service Providers. Then the external secrets operator takes care of ensuring that the credentials are populated to secrets for developers to consume. This also reduces the friction of revoking credentials as the external secrets manager (i.e., Conjur, Vault, etc.) act as a single source of truth where the DevSecOps team can invalidate credentials, as opposed to needing to manually delete secrets within a Kubernetes cluster.
Again, the result is improved security. Developers no longer have the responsibility of creating and managing secrets in the cluster, limiting their RBAC roles to the “get, list, and watch” verbs for secrets. Additionally, administrators gain additional tools to monitor and manage secrets across multiple namespaces and clusters.
A simple example
The ESO uses two custom resources (CR) to manage references to external secret management systems and references to specific secrets stored within these systems: SecretStore (or ClusterSecretStore for cluster wide scopes) and ExternalSecret (or ClusterExternalSecret), respectively. A SecretStore defines how the ESO is expected to communicate with the external secrets manager (e.g., the IP address of the server and the authentication method to use). Then the ExternalSecret can point to a SecretStore using spec.secretStoreRef to specify where it expects a certain credential is stored.
# WRITTEN BY THE DEVSECOPS TEAM
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-external-secret
namespace: eso-demo-ns
spec:
refreshPolicy: Periodic
refreshInterval: 1m
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: vault-secret-example
deletionPolicy: Delete
data:
- secretKey: password
remoteRef:
key: secret
property: passwordHere, the ExternalSecret custom resource is defined to include a creationPolicy, deletionPolicy, and a refreshPolicy (three of the knobs an administrator can use to assist in the management of secrets on their cluster).
A creationPolicy dictates how a Kubernetes secret is created from credential material originating from the secrets management database. Additionally, refreshPolicy specifies how often ESO should check for changes to the secret in the database. Finally, deletionPolicy defines what should happen to the Kubernetes secret should the upstream credential be deleted. The output of these policies are published as events on the ExternalSecret object allowing admins to easily audit or monitor credentials living in their cluster.
# WRITTEN BY THE DEVELOPER
kind: Pod
apiVersion: v1
metadata:
name: eso-demo
namespace: eso-demo-ns
spec:
containers:
- image: busybox:stable
name: eso-demo
command: ["sh", "-c", "while true; do sleep 3600; done"]
env: # Only add both env and volume clauses as an example. Use one or the other
- name: DEMO_PWD
valueFrom:
secretKeyRef:
name: vault-secret-example
key: password
volumeMounts: # Only add both env and volume clauses as an example. Use one or the other
- name: eso-demo-volume
mountPath: "/mnt/demo-vol"
readOnly: true
volumes:
- name: eso-demo-volume
secret:
secretName: vault-secret-exampleAn app developer only needs to specify one of the two highlighted clauses in their deployment script to request a secret. This means that as a developer, there is no need to ever create a Kubernetes secret.
Ultimately, the use of the external secrets operator in conjunction with an external secrets management tool improves the security posture of a cluster over a number of vectors. It limits access by shrinking the permissions scope for developers. Arguably, no human user needs to have read access to a Kubernetes secret. It improves visibility by improving the auditability of ExternalSecrets, and as a result, Kubernetes secrets living on the cluster.
Finally, it shrinks the blast radius of a breach by minimizing the impact period. Assuming a credential leak, the DevSecOps team could delete the credential in a single place (i.e., Vault), and the external secrets operator would handle the deletion of any reference to the credential automatically. Alternatively, you could update (or rotate) a credential in the database, and the external secrets operator will handle updating the credential across the cluster, minimizing downtime. Typically, rotation is considered a best practice as it minimizes downtime while still invalidating the leaked credential.
Integrations with existing Red Hat products
The external secrets operator is a highly flexible tool and enables administrators to secure secrets used for applications across the cluster. This includes the suite of layered products developed by Red Hat. Generally speaking, any workload or Red Hat product that needs credentials to operate can take advantage of secrets managed by ESO. This includes tools like Red Hat Pipelines, Red Hat OpenShift Serverless, or the newly announced Kubernetes mcp server. Next, we will discuss some of the integrations with Red Hat developed operators that we are most excited about.
GitOps with Argo CD
GitOps with Argo CD is one of the most popular tools within Red Hat’s suite of products, as it allows developers to manage their applications completely through code. The GitOps operator can connect to Git repositories to build and deploy applications in a consistent, repeatable manner. However, this means providing the operator with the necessary Git token to access one or many repositories. Rather than simply placing the Git token into a secret within the Kubernetes cluster, you can store it in an external secrets manager and allow ESO to manage the secret. As a result, all your GitOps workflows inherit the additional level of security provided by the external secrets operator.
Finally, and perhaps a more nuanced application of the external secrets operator, you can place the CRs utilized by ESO (like the ExternalSecret custom resource) in Git and manage with a GitOps workflow. This is an extremely versatile symbiotic relationship between GitOps and ESO as ESO can manage the secrets needed by applications deployed by GitOps. GitOps provides a more prescriptive pattern for change management of these resources on the OpenShift cluster.
Quay and Quay.io
This is a great example of ESO’s ability to use advanced templating to construct meaningful credentials from multiple pieces of information. The external secrets operator introduces a new CRD called a Generator, which enables developers to programmatically specify how a secret should be created given certain pieces of information. This is particularly useful in the creation of short-lived tokens such as those used as pull secrets for the Quay registry.
Quay is a highly scalable registry platform that offers multiple authentication methods to access the content it stores. Robot account federation enables developers to utilize short-lived credentials by federating with an existing OpenID Connect (OIDC) provider. An administrator can deploy a Generator, specifying simple information like the Quay URL and a service account name, and the Generator will create the necessary JWT tokens for downstream applications to consume. You can learn more about this in our comprehensive 3-part series.
Future integrations
One of the operators we are very excited to collaborate with is the Zero Trust Workload Identity Manager (ZTWIM), based on the upstream Spiffe/Spire project. ZTWIM provides secure identities for machine workloads on an OpenShift cluster. This is extremely valuable for something like the external secrets operator, which requires a more permissive set of roles to create, delete, and update secrets across a cluster.
It also requires a larger permission scope to access secrets within the external secrets management repository. This last example is a good example of a secret zero problem. If a malicious actor were to gain access to the secret zero, they'd have access to a broad scope of credentials within the secrets management repository. Leveraging ZTWIM to supply secret zero to ESO significantly helps in mitigating this attack vector.
Migrating from the upstream version
While the Red Hat version of the external secrets operator is largely a downstream fork of the community version, we've made many improvements to the security and stability of the operator. As a result, there are a couple things you should consider when preparing to migrate from the upstream community version of the operator to the Red Hat downstream published version.
We have published migration steps in the OpenShift documentation and strongly recommend you read the migration steps completely before beginning migration. Additionally, we highly recommend performing the migration during lower traffic periods or maintenance windows as the credentials managed by the operator could impact mission critical workloads.
Explore more
The general availability of the external secrets operator rounds out a trio of secrets management products offered by Red Hat. If you're interested in learning more about secrets management, take a look at our learning path on secrets management. Try the external secrets operator by checking out the OpenShift documentation and upstream documentation. You can see the complete lifecycle details, including specific dates, on the OpenShift Operator Lifecycle page.
Along with the general availability of Red Hat’s downstream version of the external secrets operator, Red Hat is excited to announce that one of our own has been nominated as an interim maintainer for the upstream project.
