Introduction
With a simple annotation to a service, you can dynamically create certificates in OpenShift.
Certificates created this way are in PEM (base64-encoded certificates) format and cannot be directly consumed by Java applications, which need certificates to be stored in Java KeyStores.
In this post, we are going to show a simple approach to enable Java applications to benefit from certificates dynamically created by OpenShift.
Why certificates
Certificates are part of a PKI infrastructure and can be used to authenticate and secure (encrypt) network communications.
OpenShift has an internal Certificate Authority (CA) that it can use to generate new certificates.
Some applications have a requirement that all communications must be encrypted, even when inside the OpenShift cluster (for example, PCI in-scope communications usually have this requirement).
Typically, in OpenShift, this use case is split into two scenarios:
- Inbound communication from outside the cluster.
- Communication between two pods running inside the cluster.
The below picture shows the two use cases:
For the route, we have chosen reencrypt so that we can use the same certificate in the server component to serve both internal and external requests and still use the OpenShift-provided automation.
If your application needs to expose its certificates directly to inbound connections then you will have to use passthrough. In this scenario, you use the ability to use the OpenShift automation.
Using this annotation in the service in front of our server pod we can have OpenShift generate certificates representing the service FQDN and put them in a secret:
service.alpha.openshift.io/serving-cert-secret-name: service-certs
Also, both the router and the consuming pod need to be able to trust the dynamically generated certificates. The route by default will trust any certificates created by the OpenShift. And for the consuming service, we can use the service account CA bundle to trust the generated certificates.
The service account CA bundle can be always be found here:
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
Unfortunately, Java applications cannot consume certificates in PEM format directly, we have to first turn them into Java Keystores.
Consuming Dynamically-Generated Certificates from Java Applications
To convert certificates in PEM format to Java KeyStores, we are going to use an init container.
The architecture and sequence of events are shown in the following picture:
We use an emptyDir volume to store the keystore and truststore files so that our application container can eventually read them.
The sequence of commands to convert a PEM-formatted certificate and private key is the following:
openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore.pkcs12 -password pass:$password keytool -importkeystore -noprompt -srckeystore $keystore.pkcs12 -srcstoretype pkcs12 -destkeystore $keystore.jks -storepass $password -srcstorepass $password
Where:
- $keyfile is the key file.
- $crtfile is the certificate file.
- $keystore_jks is the keystore file that will be created.
- $password is the password to the keystore.
- $keystore_pkcs12 is a pkcs12-formatted keystore file that is created in the process.
Our init container will look as follows:
initContainers:
- name: pem-to-keystore
image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
env:
- name: keyfile
value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
- name: crtfile
value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
- name: keystore_pkcs12
value: /var/run/secrets/java.io/keystores/keystore.pkcs12
- name: keystore_jks
value: /var/run/secrets/java.io/keystores/keystore.jks
- name: password
value: changeit
command: ['/bin/bash']
args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
volumeMounts:
- name: keystore-volume
mountPath: /var/run/secrets/java.io/keystores
- name: service-certs
mountPath: /var/run/secrets/openshift.io/services_serving_certs
volumes:
- name: keystore-volume
emptyDir: {}
- name: service-certs
secret:
secretName: service-certs
The command to create a Java Truststore starting from a CA bundle is the following:
csplit -z -f crt- service-ca.crt '/-----BEGIN CERTIFICATE-----/' '{*}'
for file in crt-*; do keytool -import -noprompt -keystore truststore.jks -file $file -storepass changeit -alias service-$file; done
Where:
- $truststore_jks is the CA bundle file.
- $ca_bundle is the generated truststore file.
- $password is the password to the truststore file.
The loop is needed because of keytool imports only one certificate at a time.
Our init container will look as follows:
initContainers:
- name: pem-to-truststore
image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
env:
- name: ca_bundle
value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
- name: truststore_jks
value: /var/run/secrets/java.io/keystores/truststore.jks
- name: password
value: changeit
command: ['/bin/bash']
args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
volumeMounts:
- name: keystore-volume
mountPath: /var/run/secrets/java.io/keystores
volumes:
- name: keystore-volume
emptyDir: {}
Note: For this example, we are using the Red Hat Single Sign-On image (version 1.1-16). This image happens to have onboard both openssl and keytool, which are the two tools that we need here. Also, RHSSO is included in any openshift subscription. You can obviously create your own image.
End-to-End SpringBoot Demo
To prove out this approach, we created a secure SpringBoot server and client that connect to it over SSL.
SSL Server
For the server, its service object will need the serving-cert-secret-name annotation to create its certificate and deployment will use the "pem-to-keystore" initContainer to create the server's keystore from the generated certificates. Below are the service, deployment config, and route definitions:
- apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.openshift.io/serving-cert-secret-name: service-certs
labels:
app: ssl-server
name: ssl-server
spec:
ports:
- name: 8443-tcp
port: 8443
protocol: TCP
targetPort: 8443
selector:
deploymentconfig: ssl-server
- apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
app: ssl-server
name: ssl-server
spec:
replicas: 1
selector:
deploymentconfig: ssl-server
template:
metadata:
labels:
app: ssl-server
deploymentconfig: ssl-server
spec:
containers:
- name: ssl-server
image: ssl-server
env:
- name: keystore_jks
value: /var/run/secrets/java.io/keystores/keystore.jks
- name: password
value: changeit
ports:
- containerPort: 8443
protocol: TCP
resources: {}
volumeMounts:
- mountPath: /var/run/secrets/java.io/keystores
name: keystore-volume
initContainers:
- name: pem-to-keystore
image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
env:
- name: keyfile
value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
- name: crtfile
value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
- name: keystore_pkcs12
value: /var/run/secrets/java.io/keystores/keystore.pkcs12
- name: keystore_jks
value: /var/run/secrets/java.io/keystores/keystore.jks
- name: password
value: changeit
command: ['/bin/bash']
args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
volumeMounts:
- mountPath: /var/run/secrets/java.io/keystores
name: keystore-volume
- mountPath: /var/run/secrets/openshift.io/services_serving_certs
name: service-certs
volumes:
- name: keystore-volume
emptyDir: {}
- name: service-certs
secret:
secretName: service-certs
- apiVersion: v1
kind: Route
metadata:
labels:
app: ssl-server
name: ssl-server
spec:
port:
targetPort: 8443-tcp
tls:
termination: reencrypt
to:
kind: Service
name: ssl-server
weight: 100
wildcardPolicy: None
We pass the keystore_jks and password values as environment variables to the app container and then in the SpringBoot application.properties file have:
server.port=8443
server.ssl.key-password=${password}
server.ssl.key-store=${keystore_jks}
server.ssl.key-store-provider=SUN
server.ssl.key-store-type=JKS
Read about configuring SSL in the SpringBoot Docs. The app has a simple /secured endpoint exposed via:
@RestController
class SecuredServerController {
@RequestMapping("/secured")
public String secured(){
System.out.println("Inside secured()");
return "Hello user !!! : " + new Date();
}
}
To start the server run:
oc new-project ssl-demo oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-server/master/template.yaml | oc create -f -
This will trigger a build and eventual deployment of the service. You can test the external route by appending /secured to the route hostname automatically generated.
SSL Client
Now for the client to make a secure connection to the server, it will need the trust store generated by the "pem-to-truststore" initContainer. Here is the client's app deployment config:
- apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
app: ssl-client
name: ssl-client
spec:
replicas: 1
selector:
deploymentconfig: ssl-client
template:
metadata:
labels:
app: ssl-client
deploymentconfig: ssl-client
spec:
containers:
- name: ssl-client
image: ssl-client
imagePullPolicy: Always
env:
- name: JAVA_OPTIONS
value: -Djavax.net.ssl.trustStore=/var/run/secrets/java.io/keystores/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
volumeMounts:
- mountPath: /var/run/secrets/java.io/keystores
name: keystore-volume
initContainers:
- name: pem-to-truststore
image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
env:
- name: ca_bundle
value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
- name: truststore_jks
value: /var/run/secrets/java.io/keystores/truststore.jks
- name: password
value: changeit
command: ['/bin/bash']
args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
volumeMounts:
- mountPath: /var/run/secrets/java.io/keystores
name: keystore-volume
volumes:
- emtpyDir: {}
name: keystore-volume
You'll note that we leveraged the JAVA_OPTIONS environment variable available on the openjdk18-openshift image to add the truststore file path and password to the image’s startup Java command.
All the client source code has are repeated calls to the server at https://ssl-client.<namespace>.svc:8443/secured
public static void main(String[] args) throws IOException, InterruptedException {
HttpClient client = new HttpClient();
GetMethod method = new GetMethod();
String uri = "https://ssl-server." + System.getenv("POD_NAMESPACE") + ".svc:8443/secured";
method.setURI(new URI(uri, false));
while(true) {
client.executeMethod(method);
Thread.sleep(5000);
}
}
To run the client app in OpenShift:
oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-client/master/template.yaml | oc create -f -
This will trigger an automatic build and deployment in your project. When the app is deployed, click on the pod logs and you should see the response from the SSL server:
Additional Findings
If your client needs the default Java CA certs as well as the CA bundle found in the pod, use this arg in the "pem-to-truststore" initContainer.
args: ['-c', "keytool -importkeystore -srckeystore $JAVA_HOME/jre/lib/security/cacerts -srcstoretype JKS -destkeystore $truststore_jks -storepass changeit -srcstorepass changeit && csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
Troubleshooting
It may be the case when working with service serving certificate secrets that you find an error annotation on the service. This means that the secret to being generated already exists. You simply need to delete the secret and recreate the service. Read the Troubleshooting Guide here.
Conclusions
This post showed a simple approach, based on init container, that allows Java applications to take advantage of OpenShift dynamically generated certificates.
Looking at Kubernetes (one of the OpenShift upstream projects) it looks like that in the future the ability of openshift to generate certificates will be improved by allowing to plugin external CAs. So, think this is a good time to start leveraging this feature also for Java applications.
To build your Java EE Microservice visit WildFly Swarm and download the cheat sheet.
Last updated: January 4, 2022