Article

Dynamically Creating Java Keystores in OpenShift

November 22, 2017
CI/CDContainersJavaSecurity
Raffaele Spazzoli
Domenic Bove
Middleware Consultant
How to customize OpenShift RBAC permissions

Introduction

With a simple annotation to a service, you can dynamically create certificates in OpenShift.

Certificates created this way are in PEM (base64-encoded certificates) format and cannot be directly consumed by Java applications, which need certificates to be stored in Java KeyStores.

In this post, we are going to show a simple approach to enable Java applications to benefit from certificates dynamically created by OpenShift.

Why certificates

Certificates are part of a PKI infrastructure and can be used to authenticate and secure (encrypt) network communications.

OpenShift has an internal Certificate Authority (CA) that it can use to generate new certificates.

Some applications have a requirement that all communications must be encrypted, even when inside the OpenShift cluster (for example, PCI in-scope communications usually have this requirement).

Typically, in OpenShift, this use case is split into two scenarios:

  1. Inbound communication from outside the cluster.
  2. Communication between two pods running inside the cluster.

The below picture shows the two use cases:

For the route, we have chosen reencrypt so that we can use the same certificate in the server component to serve both internal and external requests and still use the OpenShift-provided automation.

If your application needs to expose its certificates directly to inbound connections then you will have to use passthrough. In this scenario, you use the ability to use the OpenShift automation.

Using this annotation in the service in front of our server pod we can have OpenShift generate certificates representing the service FQDN and put them in a secret:

service.alpha.openshift.io/serving-cert-secret-name: service-certs

Also, both the router and the consuming pod need to be able to trust the dynamically generated certificates. The route by default will trust any certificates created by the OpenShift. And for the consuming service, we can use the service account CA bundle to trust the generated certificates.

The service account CA bundle can be always be found here:

/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

Unfortunately, Java applications cannot consume certificates in PEM format directly, we have to first turn them into Java Keystores.

Consuming Dynamically-Generated Certificates from Java Applications

To convert certificates in PEM format to Java KeyStores, we are going to use an init container.

The architecture and sequence of events are shown in the following picture:

We use an emptyDir volume to store the keystore and truststore files so that our application container can eventually read them.

The sequence of commands to convert a PEM-formatted certificate and private key is the following:

openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore.pkcs12 -password pass:$password
keytool -importkeystore -noprompt -srckeystore $keystore.pkcs12 -srcstoretype pkcs12 -destkeystore $keystore.jks -storepass $password -srcstorepass $password

Where:

  • $keyfile is the key file.
  • $crtfile is the certificate file.
  • $keystore_jks is the keystore file that will be created.
  • $password is the password to the keystore.
  • $keystore_pkcs12 is a pkcs12-formatted keystore file that is created in the process.

Our init container will look as follows:

  initContainers:
  - name: pem-to-keystore
  image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
  env:
    - name: keyfile
      value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
    - name: crtfile
      value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
    - name: keystore_pkcs12
      value: /var/run/secrets/java.io/keystores/keystore.pkcs12
    - name: keystore_jks
      value: /var/run/secrets/java.io/keystores/keystore.jks
    - name: password
      value: changeit    
  command: ['/bin/bash']
  args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
  volumeMounts:
    - name: keystore-volume
      mountPath: /var/run/secrets/java.io/keystores
    - name: service-certs
      mountPath: /var/run/secrets/openshift.io/services_serving_certs   
volumes:
  - name: keystore-volume
    emptyDir: {}
  - name: service-certs
    secret:
      secretName: service-certs

The command to create a Java Truststore starting from a CA bundle is the following:

csplit -z -f crt- service-ca.crt '/-----BEGIN CERTIFICATE-----/' '{*}'
for file in crt-*; do keytool -import -noprompt -keystore truststore.jks -file $file -storepass changeit -alias service-$file; done

Where:

  • $truststore_jks is the CA bundle file.
  • $ca_bundle is the generated truststore file.
  • $password is the password to the truststore file.

The loop is needed because of keytool imports only one certificate at a time.

Our init container will look as follows:

initContainers:
- name: pem-to-truststore
  image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
  env:
    - name: ca_bundle
      value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    - name: truststore_jks
      value: /var/run/secrets/java.io/keystores/truststore.jks
    - name: password
      value: changeit    
  command: ['/bin/bash']
  args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
  volumeMounts:
    - name: keystore-volume
      mountPath: /var/run/secrets/java.io/keystores  
volumes:
  - name: keystore-volume
    emptyDir: {}            

Note: For this example, we are using the Red Hat Single Sign-On image (version 1.1-16). This image happens to have onboard both openssl and keytool, which are the two tools that we need here. Also, RHSSO is included in any openshift subscription. You can obviously create your own image.

End-to-End SpringBoot Demo

To prove out this approach, we created a secure SpringBoot server and client that connect to it over SSL.

SSL Server

For the server, its service object will need the serving-cert-secret-name annotation to create its certificate and deployment will use the "pem-to-keystore" initContainer to create the server's keystore from the generated certificates. Below are the service, deployment config, and route definitions:

- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.alpha.openshift.io/serving-cert-secret-name: service-certs
    labels:
      app: ssl-server
    name: ssl-server
  spec:
    ports:
    - name: 8443-tcp
      port: 8443
      protocol: TCP
      targetPort: 8443
    selector:
      deploymentconfig: ssl-server
- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    labels:
      app: ssl-server
    name: ssl-server
  spec:
    replicas: 1
    selector:
      deploymentconfig: ssl-server
    template:
      metadata:
        labels:
          app: ssl-server
          deploymentconfig: ssl-server
      spec:
        containers:
        - name: ssl-server
          image: ssl-server
          env:
          - name: keystore_jks
            value: /var/run/secrets/java.io/keystores/keystore.jks
          - name: password
            value: changeit
          ports:
          - containerPort: 8443
            protocol: TCP
          resources: {}
          volumeMounts:
          - mountPath: /var/run/secrets/java.io/keystores
            name: keystore-volume
        initContainers:
        - name: pem-to-keystore
          image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
          env:
          - name: keyfile
            value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
          - name: crtfile
            value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
          - name: keystore_pkcs12
            value: /var/run/secrets/java.io/keystores/keystore.pkcs12
          - name: keystore_jks
            value: /var/run/secrets/java.io/keystores/keystore.jks
          - name: password
            value: changeit
          command: ['/bin/bash']
          args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
          volumeMounts:
          - mountPath: /var/run/secrets/java.io/keystores
            name: keystore-volume
          - mountPath: /var/run/secrets/openshift.io/services_serving_certs
            name: service-certs
        volumes:
        - name: keystore-volume
          emptyDir: {}
        - name: service-certs
          secret:
            secretName: service-certs
- apiVersion: v1
  kind: Route
  metadata:
    labels:
      app: ssl-server
    name: ssl-server
  spec:
    port:
      targetPort: 8443-tcp
    tls:
      termination: reencrypt
    to:
      kind: Service
      name: ssl-server
      weight: 100
    wildcardPolicy: None

We pass the keystore_jks and password values as environment variables to the app container and then in the SpringBoot application.properties file have:

server.port=8443
server.ssl.key-password=${password}
server.ssl.key-store=${keystore_jks}
server.ssl.key-store-provider=SUN
server.ssl.key-store-type=JKS

Read about configuring SSL in the SpringBoot Docs. The app has a simple /secured endpoint exposed via:

@RestController
class SecuredServerController {    
    @RequestMapping("/secured")
    public String secured(){
 System.out.println("Inside secured()");
 return "Hello user !!! : " + new Date();
    }
}

To start the server run:

oc new-project ssl-demo
oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-server/master/template.yaml | oc create -f -

This will trigger a build and eventual deployment of the service. You can test the external route by appending /secured to the route hostname automatically generated.

SSL Client

Now for the client to make a secure connection to the server, it will need the trust store generated by the "pem-to-truststore" initContainer. Here is the client's app deployment config: 

- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    labels:
      app: ssl-client
    name: ssl-client
  spec:
    replicas: 1
    selector:
      deploymentconfig: ssl-client
    template:
      metadata:
        labels:
          app: ssl-client
          deploymentconfig: ssl-client
      spec:
        containers:
        - name: ssl-client
          image: ssl-client
          imagePullPolicy: Always
          env:
          - name: JAVA_OPTIONS
            value: -Djavax.net.ssl.trustStore=/var/run/secrets/java.io/keystores/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          volumeMounts:
          - mountPath: /var/run/secrets/java.io/keystores
            name: keystore-volume
        initContainers:
        - name: pem-to-truststore
          image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
          env:
          - name: ca_bundle
            value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
          - name: truststore_jks
            value: /var/run/secrets/java.io/keystores/truststore.jks
          - name: password
            value: changeit
          command: ['/bin/bash']
          args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
          volumeMounts:
          - mountPath: /var/run/secrets/java.io/keystores
            name: keystore-volume
        volumes:
        - emtpyDir: {}
          name: keystore-volume

You'll note that we leveraged the JAVA_OPTIONS environment variable available on the openjdk18-openshift image to add the truststore file path and password to the image’s startup Java command.

All the client source code has are repeated calls to the server at https://ssl-client.<namespace>.svc:8443/secured

public static void main(String[] args) throws IOException, InterruptedException {
  HttpClient client = new HttpClient();
  GetMethod method = new GetMethod();
  String uri = "https://ssl-server." + System.getenv("POD_NAMESPACE") + ".svc:8443/secured";
  method.setURI(new URI(uri, false));
  while(true) {
    client.executeMethod(method);
    Thread.sleep(5000);
  }
}

To run the client app in OpenShift: 

oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-client/master/template.yaml | oc create -f -

This will trigger an automatic build and deployment in your project. When the app is deployed, click on the pod logs and you should see the response from the SSL server:

Additional Findings

If your client needs the default Java CA certs as well as the CA bundle found in the pod, use this arg in the "pem-to-truststore" initContainer.

args: ['-c', "keytool -importkeystore -srckeystore $JAVA_HOME/jre/lib/security/cacerts -srcstoretype JKS -destkeystore $truststore_jks -storepass changeit -srcstorepass changeit && csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]

Troubleshooting

It may be the case when working with service serving certificate secrets that you find an error annotation on the service. This means that the secret to being generated already exists. You simply need to delete the secret and recreate the service. Read the Troubleshooting Guide here.

Conclusions

This post showed a simple approach, based on init container, that allows Java applications to take advantage of OpenShift dynamically generated certificates.

Looking at Kubernetes (one of the OpenShift upstream projects) it looks like that in the future the ability of openshift to generate certificates will be improved by allowing to plugin external CAs. So, think this is a good time to start leveraging this feature also for Java applications.

To build your Java EE Microservice visit WildFly Swarm and download the cheat sheet.

Last updated: January 4, 2022