This is the third installment of our series about zero-trust patterns with Red Hat Quay leveraging short-lived credentials within Red Hat OpenShift. In part 1, we discussed how short-lived credentials in Quay improve security.
In part 2, we introduced the concept of federated robot accounts in Quay that can authenticate using short-lived credentials. We demonstrated how a JSON Web Token (JWT) issued by a trusted OpenID Connect (OIDC) provider enables federated authentication, using robot accounts within Quay. We also explored two methods of exchanging a JWT for credentials to access Quay content from within an OpenShift workload. While each satisfied their desired function, the process of retrieving the JWT and facilitating the exchange with Quay was manual in nature. But we did not address the need to automatically refresh the credentials once they expire and how this may impact the workload trying to communicate with Quay.
Automating the process
Within any cloud native environment, there is always the desire to automate as much as possible and reduce or eliminate any manual actions. Likewise, we ideally want the cluster’s communication with the registry, which is a prerequisite for starting any workload pods, to leverage short-lived credentials too.
One such solution that fulfills the goal of automating the process of obtaining short-lived credentials from Quay is the External Secrets Operator. By once again leveraging the service account token federation capability of OpenShift, the entire process of performing the initial exchange as well as subsequently refreshing the short-lived credentials is automated, and the resulting Quay credentials are stored in a Secret within the cluster.
Let’s demonstrate how to integrate the External Secrets Operator to manage the lifecycle of Quay credentials at the Kubernetes level. Since the External Secrets Operator relies on service account token federation, the use of this feature can only be used in environments that support this capability. Refer to the prior article that focused on the OpenShift service account token federation section for the list of suitable environments. In addition, since the External Secrets Operator makes use of several Custom Resources, elevated permissions to the OpenShift environment is required in order to install the associated Custom Resource Definitions (CRDs).
The deployment of the External Secrets Operator to OpenShift is facilitated using Helm. Execute the following commands to add the Helm chart repository and install the operator to a namespace called external-secrets within the cluster:
# Add the external-secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
# Install the External Secrets Helm chart
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=trueWith the operator installed, let's explore how the External Secrets Operator integrates with Quay.
An ExternalSecret is the primary resource provided by the External Secrets Operator. It describes where the secure resource should be obtained from and how the contents should be stored in the resulting Secret within the cluster. The integration with Quay leverages a generator called QuayAccessToken which contains the necessary information to perform the exchange of a JWT with Quay in a similar fashion as was performed manually.
The same namespace (quay-keyless) and service account (quay-keyless) from the previous articles will be reused again within this section. Refer to the OpenShift service account token federation section in these posts for the steps to create these resources in the event they do not currently exist.
Start by creating several environment variables based on the contents within the OpenShift cluster and the target Quay instance.
Set the QUAY_REGISTRY_HOSTNAME environment variable representing the hostname of the Quay registry instance. For example, if quay.io is the desired target instance, the value of the QUAY_REGISTRY_HOSTNAME variable would be quay.io:
export QUAY_REGISTRY_HOSTNAME=<quay_registry> Next, set the full name of the robot account in a variable called ROBOT_ACCOUNT. The full name of a robot account is represented by the organization for which the robot account is part of and the name of the robot account, separated by a plus (+) sign. For the robot account previously created, the full name is <organization>+keyless:
export ROBOT_ACCOUNT=<full_robot_account_name>Now, create the QuayAccessToken Custom Resource by rendering the content using the following command:
cat <<EOF | oc apply -f -
apiVersion: generators.external-secrets.io/v1alpha1
kind: QuayAccessToken
metadata:
name: quay-keyless-token
namespace: quay-keyless
spec:
robotAccount: ${ROBOT_ACCOUNT}
serviceAccountRef:
name: quay-keyless
namespace: quay-keyless
url: ${QUAY_REGISTRY_HOSTNAME}
EOFBefore creating the ExternalSecret Custom Resource, ensure that the Quay robot account has been configured to be usable with the OpenShift service account. This requires that the content of issuer and subject fields of a JWT generated from that service account are registered with the robot account in Quay in order to establish mutual trust with the signing OIDC provider.
To check the values of the OIDC Issuer and Subject, generate a new JWT for the service account using the oc create token command and store the resulting value in a variable called SA_JWT:
SA_JWT=$(oc create token -n quay-keyless quay-keyless)Obtain the OIDC issuer from the token:
echo $SA_JWT | cut -d '.' -f 2 | sed 's/[^=]$/&==/' | base64 -d | jq -r '.iss'Do the same for the subject:
echo $SA_JWT | cut -d '.' -f 2 | sed 's/[^=]$/&==/' | base64 -d | jq -r '.sub'If the federation configuration from the prior article is still associated with the robot account, no additional action is needed. Otherwise, navigate to the Robot accounts tab where the keyless robot account is located, click the menu kebab next to the keyless robot account, and select Set robot federation, as shown in Figure 1.
Click the plus (+) button and enter the OIDC issuer and subject that was retrieved previously from the JWT within the Pod. Click Save to apply the changes and then Close to minimize the dialog (Figure 2).
At this point, we have the External Secrets Operator regularly fetching short-lived credentials from our federated robot account in Quay, based on the service account token from the trusted OIDC Provider. However, this isn’t usable yet at the Kubernetes level.
For that, let's turn our attention to the ExternalSecret resource that will be used to populate a Secret with these short-lived Quay credentials, so we can refer to it as an image pull secret. This can be achieved with an ExternalSecret which can set individual keys based on remote references or render content using a robust templating engine.
In this instance, we will generate a Secret of type kubernetes.io/dockerconfigjson with a template containing contents provided by the QuayAccessToken previously created. Three values are available for use from the QuayAccessToken generator within an ExternalSecret:
- registry: Domain name of the quay registry.
- auth: Authentication string in base64 encoded format.
- expires: Time when the credentials expire (in UNIX time).
To render a dockerconfigjson formatted Secret using the aforementioned values, we can use a template similar to the following:
template:
data:
.dockerconfigjson: |
{
"auths": {
"{{ .registry }}": {
"auth": "{{ .auth }}"
}
}
}Now, create the ExternalSecret using the following command:
cat <<EOF | oc apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: quay-credentials
namespace: quay-keyless
spec:
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: QuayAccessToken
name: quay-keyless-token
refreshInterval: 55m
target:
creationPolicy: Owner
deletionPolicy: Retain
name: quay-registry-credentials
template:
data:
.dockerconfigjson: |
{
"auths": {
"{{ .registry }}": {
"auth": "{{ .auth }}"
}
}
}
engineVersion: v2
mergePolicy: Replace
type: kubernetes.io/dockerconfigjson
EOFTake note of the reference to the QuayAccessToken in the .spec.dataFrom property, as it represents the resource providing the content to the ExternalSecret. Recall from the prior article that the credentials provided by Quay have a lifespan of 1 hour. As a result, the refreshInterval property has a value of 55 minutes, so that a new set of credentials will be retrieved prior to the expiration time.
The .spec.target property contains information related to the Secret that will be created and its contents. The Secret that will be created will be called quay-registry-credentials as denoted by the name property within the same namespace as the ExternalSecret.
Confirm that the ExternalSecret was synced successfully:
oc get externalsecret -n quay-keyless
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY
quay-credentials 55m SecretSynced TrueIf the ExternalSecret has a status of SecretSynced, locate the quay-registry-credentials Secret that was generated in the quay-keyless namespace and describe the contents:
oc describe secret -n quay-keyless quay-registry-credentials
Name: quay-registry-credentials
Namespace: quay-keyless
Labels: reconcile.external-secrets.io/created-by=e40de526b46e03267ddd234ad1fd714d
reconcile.external-secrets.io/managed=true
Annotations: reconcile.external-secrets.io/data-hash: f3a51d00a8f7dc7275e8a15f67f360d9
Type: kubernetes.io/dockerconfigjson
Data
====
.dockerconfigjson: 1388 bytesWith the content in dockerconfigjson format, the values can be used in a variety of ways. For example, the Secret could be mounted to a pod so that it can be leveraged by a container engine, like Podman, to access content within the Quay registry.
The most common use of a Secret containing registry credentials in this format is that it can be used as an Image Pull Secret enabling a workload to make use of a protected image. This can be seen in the following example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: image-pull-secret-example
namespace: quay-keyless
labels:
app: image-pull-secret-example
spec:
replicas: 1
selector:
matchLabels:
app: image-pull-secret-example
template:
metadata:
labels:
app: image-pull-secret-example
spec:
imagePullSecrets:
- name: quay-registry-credentials
containers:
- name: image-pull-secret-example
image: quay.example.com/quay_federation/myimage:latestBe sure to update the image field with the location of a protected image sourced from your Quay registry.
Aside from setting the imagePullSecret property of a workload, a Secret can be added to a service account within the imagePullSecrets property. When the associated service account is specified against a workload, the imagePullSecret property of the Pod will be automatically set.
Going beyond the basics, one final use case for which the Secret can be used is within a Tekton (Red Hat OpenShift Pipelines) pipeline. In order for the credential to be eligible for use within a pipeline, it must be annotated with the tekton.dev/docker-* annotation containing the URL of the registry. The template within the ExternalSecret can be updated with the following:
template:
metadata:
annotations:
tekton.dev/docker-0: https://{{ .registry }}
data:
.dockerconfigjson: |
{
"auths": {
"{{ .registry }}": {
"auth": "{{ .auth }}"
}
}
}By describing the Secret using the command previously provided, you should see the updated annotation with the URL of the Quay registry enabling its use within Tekton.
Final thoughts
In this final article in our series, we described how one can easily leverage the short-lived credentials of federated robot tokens in Quay with existing Kubernetes workloads. In combination with the previous articles in this series, we have introduced several different approaches, including frameworks that support supplying identities to workloads (i.e., SPIFE/SPIRE) and tools that automate the lifecycle of credentials (External Secrets Operator). We hope that you will investigate their use within your own environment to strengthen the security posture when interacting with Quay.
The External Secrets Operator is in the productized process in OpenShift so we can offer it as a supported capability for all OpenShift customers. Stay tuned for further announcements in this space.
