Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Dynamically Creating Java Keystores in OpenShift

November 22, 2017
Raffaele Spazzoli Domenic Bove
Related topics:
CI/CDContainersJavaSecurity
Related products:
Red Hat JBoss Enterprise Application Platform

Share:

    Introduction

    With a simple annotation to a service, you can dynamically create certificates in OpenShift.

    Certificates created this way are in PEM (base64-encoded certificates) format and cannot be directly consumed by Java applications, which need certificates to be stored in Java KeyStores.

    In this post, we are going to show a simple approach to enable Java applications to benefit from certificates dynamically created by OpenShift.

    Why certificates

    Certificates are part of a PKI infrastructure and can be used to authenticate and secure (encrypt) network communications.

    OpenShift has an internal Certificate Authority (CA) that it can use to generate new certificates.

    Some applications have a requirement that all communications must be encrypted, even when inside the OpenShift cluster (for example, PCI in-scope communications usually have this requirement).

    Typically, in OpenShift, this use case is split into two scenarios:

    1. Inbound communication from outside the cluster.
    2. Communication between two pods running inside the cluster.

    The below picture shows the two use cases:

    For the route, we have chosen reencrypt so that we can use the same certificate in the server component to serve both internal and external requests and still use the OpenShift-provided automation.

    If your application needs to expose its certificates directly to inbound connections then you will have to use passthrough. In this scenario, you use the ability to use the OpenShift automation.

    Using this annotation in the service in front of our server pod we can have OpenShift generate certificates representing the service FQDN and put them in a secret:

    service.alpha.openshift.io/serving-cert-secret-name: service-certs

    Also, both the router and the consuming pod need to be able to trust the dynamically generated certificates. The route by default will trust any certificates created by the OpenShift. And for the consuming service, we can use the service account CA bundle to trust the generated certificates.

    The service account CA bundle can be always be found here:

    /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

    Unfortunately, Java applications cannot consume certificates in PEM format directly, we have to first turn them into Java Keystores.

    Consuming Dynamically-Generated Certificates from Java Applications

    To convert certificates in PEM format to Java KeyStores, we are going to use an init container.

    The architecture and sequence of events are shown in the following picture:

    We use an emptyDir volume to store the keystore and truststore files so that our application container can eventually read them.

    The sequence of commands to convert a PEM-formatted certificate and private key is the following:

    openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore.pkcs12 -password pass:$password
    keytool -importkeystore -noprompt -srckeystore $keystore.pkcs12 -srcstoretype pkcs12 -destkeystore $keystore.jks -storepass $password -srcstorepass $password

    Where:

    • $keyfile is the key file.
    • $crtfile is the certificate file.
    • $keystore_jks is the keystore file that will be created.
    • $password is the password to the keystore.
    • $keystore_pkcs12 is a pkcs12-formatted keystore file that is created in the process.

    Our init container will look as follows:

      initContainers:
      - name: pem-to-keystore
      image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
      env:
        - name: keyfile
          value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
        - name: crtfile
          value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
        - name: keystore_pkcs12
          value: /var/run/secrets/java.io/keystores/keystore.pkcs12
        - name: keystore_jks
          value: /var/run/secrets/java.io/keystores/keystore.jks
        - name: password
          value: changeit    
      command: ['/bin/bash']
      args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
      volumeMounts:
        - name: keystore-volume
          mountPath: /var/run/secrets/java.io/keystores
        - name: service-certs
          mountPath: /var/run/secrets/openshift.io/services_serving_certs   
    volumes:
      - name: keystore-volume
        emptyDir: {}
      - name: service-certs
        secret:
          secretName: service-certs 

    The command to create a Java Truststore starting from a CA bundle is the following:

    csplit -z -f crt- service-ca.crt '/-----BEGIN CERTIFICATE-----/' '{*}'
    for file in crt-*; do keytool -import -noprompt -keystore truststore.jks -file $file -storepass changeit -alias service-$file; done

    Where:

    • $truststore_jks is the CA bundle file.
    • $ca_bundle is the generated truststore file.
    • $password is the password to the truststore file.

    The loop is needed because of keytool imports only one certificate at a time.

    Our init container will look as follows:

    initContainers:
    - name: pem-to-truststore
      image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
      env:
        - name: ca_bundle
          value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        - name: truststore_jks
          value: /var/run/secrets/java.io/keystores/truststore.jks
        - name: password
          value: changeit    
      command: ['/bin/bash']
      args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
      volumeMounts:
        - name: keystore-volume
          mountPath: /var/run/secrets/java.io/keystores  
    volumes:
      - name: keystore-volume
        emptyDir: {}            

    Note: For this example, we are using the Red Hat Single Sign-On image (version 1.1-16). This image happens to have onboard both openssl and keytool, which are the two tools that we need here. Also, RHSSO is included in any openshift subscription. You can obviously create your own image.

    End-to-End SpringBoot Demo

    To prove out this approach, we created a secure SpringBoot server and client that connect to it over SSL.

    SSL Server

    For the server, its service object will need the serving-cert-secret-name annotation to create its certificate and deployment will use the "pem-to-keystore" initContainer to create the server's keystore from the generated certificates. Below are the service, deployment config, and route definitions:

    - apiVersion: v1
      kind: Service
      metadata:
        annotations:
          service.alpha.openshift.io/serving-cert-secret-name: service-certs
        labels:
          app: ssl-server
        name: ssl-server
      spec:
        ports:
        - name: 8443-tcp
          port: 8443
          protocol: TCP
          targetPort: 8443
        selector:
          deploymentconfig: ssl-server
    - apiVersion: v1
      kind: DeploymentConfig
      metadata:
        labels:
          app: ssl-server
        name: ssl-server
      spec:
        replicas: 1
        selector:
          deploymentconfig: ssl-server
        template:
          metadata:
            labels:
              app: ssl-server
              deploymentconfig: ssl-server
          spec:
            containers:
            - name: ssl-server
              image: ssl-server
              env:
              - name: keystore_jks
                value: /var/run/secrets/java.io/keystores/keystore.jks
              - name: password
                value: changeit
              ports:
              - containerPort: 8443
                protocol: TCP
              resources: {}
              volumeMounts:
              - mountPath: /var/run/secrets/java.io/keystores
                name: keystore-volume
            initContainers:
            - name: pem-to-keystore
              image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
              env:
              - name: keyfile
                value: /var/run/secrets/openshift.io/services_serving_certs/tls.key
              - name: crtfile
                value: /var/run/secrets/openshift.io/services_serving_certs/tls.crt
              - name: keystore_pkcs12
                value: /var/run/secrets/java.io/keystores/keystore.pkcs12
              - name: keystore_jks
                value: /var/run/secrets/java.io/keystores/keystore.jks
              - name: password
                value: changeit
              command: ['/bin/bash']
              args: ['-c', "openssl pkcs12 -export -inkey $keyfile -in $crtfile -out $keystore_pkcs12 -password pass:$password && keytool -importkeystore -noprompt -srckeystore $keystore_pkcs12 -srcstoretype pkcs12 -destkeystore $keystore_jks -storepass $password -srcstorepass $password"]
              volumeMounts:
              - mountPath: /var/run/secrets/java.io/keystores
                name: keystore-volume
              - mountPath: /var/run/secrets/openshift.io/services_serving_certs
                name: service-certs
            volumes:
            - name: keystore-volume
              emptyDir: {}
            - name: service-certs
              secret:
                secretName: service-certs
    - apiVersion: v1
      kind: Route
      metadata:
        labels:
          app: ssl-server
        name: ssl-server
      spec:
        port:
          targetPort: 8443-tcp
        tls:
          termination: reencrypt
        to:
          kind: Service
          name: ssl-server
          weight: 100
        wildcardPolicy: None

    We pass the keystore_jks and password values as environment variables to the app container and then in the SpringBoot application.properties file have:

    server.port=8443
    server.ssl.key-password=${password}
    server.ssl.key-store=${keystore_jks}
    server.ssl.key-store-provider=SUN
    server.ssl.key-store-type=JKS 

    Read about configuring SSL in the SpringBoot Docs. The app has a simple /secured endpoint exposed via:

    @RestController
    class SecuredServerController {    
        @RequestMapping("/secured")
        public String secured(){
     System.out.println("Inside secured()");
     return "Hello user !!! : " + new Date();
        }
    }

    To start the server run:

    oc new-project ssl-demo
    oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-server/master/template.yaml | oc create -f -

    This will trigger a build and eventual deployment of the service. You can test the external route by appending /secured to the route hostname automatically generated.

    SSL Client

    Now for the client to make a secure connection to the server, it will need the trust store generated by the "pem-to-truststore" initContainer. Here is the client's app deployment config:

    - apiVersion: v1
      kind: DeploymentConfig
      metadata:
        labels:
          app: ssl-client
        name: ssl-client
      spec:
        replicas: 1
        selector:
          deploymentconfig: ssl-client
        template:
          metadata:
            labels:
              app: ssl-client
              deploymentconfig: ssl-client
          spec:
            containers:
            - name: ssl-client
              image: ssl-client
              imagePullPolicy: Always
              env:
              - name: JAVA_OPTIONS
                value: -Djavax.net.ssl.trustStore=/var/run/secrets/java.io/keystores/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              volumeMounts:
              - mountPath: /var/run/secrets/java.io/keystores
                name: keystore-volume
            initContainers:
            - name: pem-to-truststore
              image: registry.access.redhat.com/redhat-sso-7/sso71-openshift:1.1-16
              env:
              - name: ca_bundle
                value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
              - name: truststore_jks
                value: /var/run/secrets/java.io/keystores/truststore.jks
              - name: password
                value: changeit
              command: ['/bin/bash']
              args: ['-c', "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]
              volumeMounts:
              - mountPath: /var/run/secrets/java.io/keystores
                name: keystore-volume
            volumes:
            - emtpyDir: {}
              name: keystore-volume

    You'll note that we leveraged the JAVA_OPTIONS environment variable available on the openjdk18-openshift image to add the truststore file path and password to the image’s startup Java command.

    All the client source code has are repeated calls to the server at https://ssl-client.<namespace>.svc:8443/secured

    public static void main(String[] args) throws IOException, InterruptedException {
      HttpClient client = new HttpClient();
      GetMethod method = new GetMethod();
      String uri = "https://ssl-server." + System.getenv("POD_NAMESPACE") + ".svc:8443/secured";
      method.setURI(new URI(uri, false));
      while(true) {
        client.executeMethod(method);
        Thread.sleep(5000);
      }
    }

    To run the client app in OpenShift:

    oc process -f https://raw.githubusercontent.com/domenicbove/openshift-ssl-client/master/template.yaml | oc create -f -

    This will trigger an automatic build and deployment in your project. When the app is deployed, click on the pod logs and you should see the response from the SSL server:

    Additional Findings

    If your client needs the default Java CA certs as well as the CA bundle found in the pod, use this arg in the "pem-to-truststore" initContainer.

    args: ['-c', "keytool -importkeystore -srckeystore $JAVA_HOME/jre/lib/security/cacerts -srcstoretype JKS -destkeystore $truststore_jks -storepass changeit -srcstorepass changeit && csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done"]

    Troubleshooting

    It may be the case when working with service serving certificate secrets that you find an error annotation on the service. This means that the secret to being generated already exists. You simply need to delete the secret and recreate the service. Read the Troubleshooting Guide here.

    Conclusions

    This post showed a simple approach, based on init container, that allows Java applications to take advantage of OpenShift dynamically generated certificates.

    Looking at Kubernetes (one of the OpenShift upstream projects) it looks like that in the future the ability of openshift to generate certificates will be improved by allowing to plugin external CAs. So, think this is a good time to start leveraging this feature also for Java applications.


    To build your Java EE Microservice visit WildFly Swarm and download the cheat sheet.

    Last updated: January 4, 2022

    Recent Posts

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    • How to integrate vLLM inference into your macOS and iOS apps

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue