In Red Hat OpenShift Container Platform (RHOCP), a robust Public Key Infrastructure (PKI) managed by various operators secures internal communication. One critical component is the node-system-admin-client certificate, which is used by the system:admin user. This user possesses the cluster-admin role, granting it the highest level of privileges required for cluster administration and maintenance tasks.
This article provides a detailed analysis of this certificate's lifecycle, addressing a common point of confusion: the discrepancy between its configured validity period in the source code and its actual expiration date. We will explore the certificate chain, investigate the relevant operator source code, and simulate a manual rotation to understand the underlying mechanics.
Certificate rotation
We will refer to OpenShift 4.16 in this article. For a freshly installed OpenShift cluster, Figure 1 illustrates the certificate rotation for node-system-admin-client.
Figure 2 illustrates an existing OCP cluster after rotation.
Certificate and signer inspection
Upon inspecting a cluster, we can observe the details of the node-system-admin-client certificate and its signer.
First, let's examine the client certificate stored in a secret within the openshift-kube-apiserver-operator namespace.
# Inspect the client certificate details
oc get secret -n openshift-kube-apiserver-operator node-system-admin-client -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -noout -issuer -subject -dates
# issuer=CN=openshift-kube-apiserver-operator_node-system-admin-signer@1752634724
# subject=O=system:masters, CN=system:admin
# notBefore=Jul 16 02:58:51 2025 GMT
# notAfter=Jul 16 02:58:44 2026 GMTNext, we inspect its issuer, the node-system-admin-signer certificate:
# Inspect the signer (CA) certificate details
oc get secret -n openshift-kube-apiserver-operator node-system-admin-signer -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -noout -issuer -subject -dates
# issuer=CN=openshift-kube-apiserver-operator_node-system-admin-signer@1752634724
# subject=CN=openshift-kube-apiserver-operator_node-system-admin-signer@1752634724
# notBefore=Jul 16 02:58:43 2025 GMT
# notAfter=Jul 16 02:58:44 2026 GMTThe output clearly shows that both the client certificate and its signer have a validity of one year. To confirm the integrity of the chain, we can perform a verification:
# Step 1: Extract the client certificate
oc get secret -n openshift-kube-apiserver-operator node-system-admin-client -o jsonpath='{.data.tls\.crt}' | base64 --decode > client.crt
# Step 2: Extract the signer's root certificate
oc get secret -n openshift-kube-apiserver-operator node-system-admin-signer -o jsonpath='{.data.tls\.crt}' | base64 --decode > root-for-csr-signer.crt
# Step 3: Verify the certification chain
openssl verify -CAfile root-for-csr-signer.crt client.crt
# client.crt: OKThe successful verification confirms that root-for-csr-signer.crt is the direct issuer of client.crt.
Investigating the validity discrepancy
A review of the cluster-kube-apiserver-operator source code reveals that the node-system-admin-client certificate is intended to have a two-year validity period.
certrotation.RotatedSelfSignedCertKeySecret{
Namespace: operatorclient.OperatorNamespace,
Name: "node-system-admin-client",
// ... annotations ...
// This needs to live longer then control plane certs so there is high chance that if a cluster breaks
// because of expired certs these are still valid to use for collecting data using localhost-recovery
// endpoint with long lived serving certs for localhost.
Validity: 2 * 365 * defaultRotationDay, // Configured for 2 years
// We rotate sooner so certs are always valid for 90 days (30 days more then kube-control-plane-signer)
Refresh: 30 * defaultRotationDay,
// ... other parameters ...
},The code comment indicates a clear intention: this certificate should be long-lived to aid in cluster recovery scenarios where other control plane certificates might have expired. So, why do we observe only a one-year validity?
The answer lies in a fundamental principle of PKI: a leaf certificate's validity period cannot extend beyond that of its signing CA.
By examining the source code for the node-system-admin-signer, we find its configuration:
certrotation.RotatedSigningCASecret{
Namespace: operatorclient.OperatorNamespace,
Name: "node-system-admin-signer",
// ... annotations ...
Validity: 1 * 365 * defaultRotationDay, // Configured for 1 year
// Refresh set to 80% of the validity.
// This range is consistent with most other signers defined in this pkg.
Refresh: 292 * defaultRotationDay,
// ... other parameters ...
},The signer CA itself is configured with a one-year validity. Therefore, even though the node-system-admin-client certificate requests a two-year lifespan, it is ultimately constrained by its issuer's one-year expiration date.
Simulating the signer CA rotation
To further understand the relationship between the signer and the client certificate, we can simulate a manual rotation of the signer CA. The following steps demonstrate what happens when renewing the CA.
Note: In a live cluster, the operator automatically handles the certificate rotation based on the Refresh interval. The manual deletion of a secret is a disruptive action used here for demonstration purposes. The actual rotation process is more graceful and includes mechanisms to ensure trust is maintained by bundling old and new CAs.
Step 1: Capture the initial state
First, we will document the creation timestamps and serial numbers of the existing signer and client certificates.
The signer CA details:
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.metadata.creationTimestamp}{"\n"}'
# 2025-07-08T22:40:46Z
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial -issuer -subject -dates
# serial=038106BD877BDCBE
# issuer=CN = openshift-kube-apiserver-operator_node-system-admin-signer@1752014437
# ...The following shows the client certificate details:
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.metadata.creationTimestamp}{"\n"}'
# 2025-07-08T22:40:53Z
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial -issuer -subject -dates
# serial=023EE1F3311AF1D5
# issuer=CN = openshift-kube-apiserver-operator_node-system-admin-signer@1752014437
# ...
Step 2: Manually trigger rotation
We can force the operator to regenerate the signer CA by deleting its corresponding secret.
oc delete secret node-system-admin-signer -n
openshift-kube-apiserver-operatorStep 3: Observe the aftermath
The operator will quickly create a new signer secret. Let's inspect it.
The following is the new signer CA details (updated):
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.metadata.creationTimestamp}{"\n"}'
# 2025-07-17T06:30:49Z
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial -issuer -subject -dates
# serial=31A8CCE1005C0642
# issuer=CN = openshift-kube-apiserver-operator_node-system-admin-signer@1752733849
# ...Now, let's check the client certificate. The following shows the client certificate details (not updated):
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.metadata.creationTimestamp}{"\n"}'
# 2025-07-08T22:40:53Z
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial -issuer -subject -dates
# serial=023EE1F3311AF1D5
# issuer=CN = openshift-kube-apiserver-operator_node-system-admin-signer@1752014437
# ...This is a critical observation: rotating the signer CA does not automatically trigger the renewal of the leaf certificates it has issued. The old client certificate remains untouched.
Step 4: Verify the broken chain
If we try to verify the old client certificate against the new CA, the validation fails as expected.
# Extract the new CA certificate
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d > new-ca.crt
# Extract the (old) leaf certificate
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d > old-leaf.crt
# Verify the chain
openssl verify -CAfile new-ca.crt old-leaf.crt
# O = system:masters, CN = system:admin
# error 20 at 0 depth lookup: unable to get local issuer certificate
# error old-leaf.crt: verification failedThe "unable to get local issuer certificate" error confirms that the issuer of old-leaf.crt is not new-ca.crt.
Forcing the client certificate renewal
To fix the broken chain, we must force the node-system-admin-client certificate to be renewed and signed by the new CA. This can be done by removing the auth.openshift.io/certificate-not-after annotation from the secret, which signals the operator to regenerate it.
oc patch secret node-system-admin-client -n openshift-kube-apiserver-operator -p='{"metadata": {"annotations": {"auth.openshift.io/certificate-not-after": null}}}'After patching, the operator re-issues the client certificate. Let's inspect the new details:
# Get serial number and issuer of the renewed client certificate
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -serial -issuer -subject -dates
# serial=028AC59829CF3F15
# issuer=CN = openshift-kube-apiserver-operator_node-system-admin-signer@1752733849
# subject=O = system:masters, CN = system:admin
# notBefore=Jul 17 06:52:17 2025 GMT
# notAfter=Jul 17 06:30:49 2026 GMTThe issuer now matches the new signer CA. Finally, we can verify the complete, valid chain.
# Extract the new CA certificate
oc get secret node-system-admin-signer -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d > new-ca.crt
# Extract the new leaf certificate
oc get secret node-system-admin-client -n openshift-kube-apiserver-operator -o=jsonpath='{.data.tls\.crt}' | base64 -d > new-leaf.crt
# Verify the chain
openssl verify -CAfile new-ca.crt new-leaf.crt
# new-leaf.crt: OKWrap up
This article provides an in-depth analysis of the node-system-admin-client certificate lifecycle in OpenShift, focusing on the discrepancy between its configured two-year validity and its actual one-year expiration. The analysis reveals that a leaf certificate's validity is constrained by its signing Certificate Authority (CA), which in this case has a one-year lifespan. The article further demonstrates that rotating the signer CA does not automatically renew issued leaf certificates, and manual intervention is required to force client certificate renewal by manipulating annotations on the certificate's secret. This highlights key aspects of OpenShift's internal PKI management, including issuer constraints on validity, the actual lifespan of the node-system-admin-client certificate, rotation mechanics, and manual renewal processes.
In summary, we demonstrated the following key aspects of the OpenShift internal PKI management.
- Issuer constrains validity: The effective validity of a certificate is capped by the expiration date of its signing CA.
- node-system-admin-client lifespan: While configured for two years for recovery purposes, its actual lifespan is one year due to its signer's configuration.
- Rotation mechanics: The rotation of a signing CA does not automatically trigger the renewal of all certificates it has issued. Leaf certificates are renewed based on their own lifecycle triggers.
- Manual renewal: Certificate renewal can be forced by manipulating annotations on the certificate's secret, signaling the managing operator to take action.
