Set up Red Hat AMQ Streams custom certificates on OpenShift (update)

April 1, 2020
Federico Valeri
Related topics:
Event-drivenJavaKubernetes
Related products:
Red Hat OpenShift

As anticipated in the "Additional notes" section of my previous article, starting from Red Hat AMQ Streams 1.4, it is finally possible to use your own custom certificate for encrypting communication between Kafka clients and brokers—without the requirement to provide a CA certificate. The auto-generated and -managed internal CAs will still remain, but only to protect inter-cluster communication.

The user-provided certificate can be used with all listeners that have TLS encryption enabled, such as the route, load balancer, ingress, and NodePort types. In this complete example, we will enable an external route listener for one-way TLS authentication.

Prerequisites

You need to have the following in place before you can proceed:

The procedure

Before we start, let's define a few handy variables:

$ USER="developer"
$ PROJECT="streams"
$ CA_USER="system:admin"
$ RA_SECRET="reg-auth-secret"
$ CLUSTER="my-cluster"

The first step is to log in as cluster-admin and create a new project. We need this role because we have to install the custom resource definitions (CRDs) that are required by the Cluster Operator (CO). We then give full admin rights to the user for managing the project once it's ready:

$ oc login -u $CA_USER
$ oc new-project $PROJECT
$ oc adm policy add-role-to-user admin $USER

To be able to download images from the Red Hat Container Registry, we also need to add an authentication Secret (use your credentials here):

$ oc create secret docker-registry $RA_SECRET \
    --docker-server=registry.redhat.io \
    --docker-username= \
    --docker-password=

Then, unzip the installation and examples distribution package (with the name ending in -install-examples.zip) and replace the default project’s name with yours:

$ TMP="/tmp/$PROJECT" && rm -rf $TMP && mkdir -p $TMP
$ unzip -qq amq-streams-1.4.0-ocp-install-examples.zip -d $TMP
$ sed -i -e "s/namespace: .*/namespace: $PROJECT/g" $TMP/install/cluster-operator/*RoleBinding*.yaml

Now, we are ready to install all required CRDs and the Strimzi CO:

$ oc apply -f $TMP/install/cluster-operator
$ oc secrets link strimzi-cluster-operator $RA_SECRET --for=pull
$ oc set env deploy/strimzi-cluster-operator STRIMZI_IMAGE_PULL_SECRETS=$RA_SECRET

$ oc set env deploy/strimzi-cluster-operator STRIMZI_NAMESPACE=$PROJECT
$ oc apply -f $install_dir/strimzi-admin
$ oc adm policy add-cluster-role-to-user strimzi-admin $USER

Test cluster creation

Here we create a small test cluster with a topic just for the sake of this example (this cluster is not suitable for production):

$ oc create -f - <<EOF
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: "2.3.1"
    replicas: 3
    config:
      log.message.format.version: "2.3"
    logging:
      type: inline
      loggers:
        log4j.logger.kafka.controller: INFO
        log4j.logger.kafka.authorizer.logger: INFO
    listeners:
      plain: {}
      external:
        type: route
    readinessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    livenessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    template:
        pod:
          terminationGracePeriodSeconds: 120
    storage:
      type: persistent-claim
      size: "1Gi"
    resources:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        cpu: "1000m"
        memory: "2Gi"
    tlsSidecar:
      resources:
        limits:
          cpu: "100m"
          memory: "128Mi"
        requests:
          cpu: "100m"
          memory: "128Mi"
  zookeeper:
    replicas: 3
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: "1Gi"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "500m"
        memory: "1Gi"
    tlsSidecar:
      resources:
        limits:
          cpu: "100m"
          memory: "128Mi"
        requests:
          cpu: "100m"
          memory: "128Mi"
  entityOperator:
    topicOperator:
      resources:
        limits:
          cpu: "250m"
          memory: "256Mi"
        requests:
          cpu: "250m"
          memory: "256Mi"
    userOperator:
      resources:
        limits:
          cpu: "250m"
          memory: "256Mi"
        requests:
          cpu: "250m"
          memory: "256Mi"
    tlsSidecar:
      resources:
        limits:
          cpu: "100m"
          memory: "128Mi"
        requests:
          cpu: "100m"
          memory: "128Mi"
EOF

After running the previous command, wait for the cluster to be up and running.

Custom certificate configuration

At this point, you should already have the following files:

  • rootca.pem - Root Certificate Authority (CA) of your domain (optional).
  • intermca.pem - Intermediate CA used to sign sub-domain certs (optional).
  • server.pem - Custom certificate to use for the external route listener.
  • server-prk.pem - The private key of custom certificate.

As we will see, if you are not using a self-signed certificate, then you can provide a certificate that includes the whole chain of trust (e.g., rootca + intermca + server).

The most important point to remember here is that your custom certificate must include the correct Subject Alternative Names (SANs). This means having one entry for the bootstrap route and one entry for each broker. You can easily find these by looking at the route's HOST/PORT column:

$ oc get routes
NAME                         HOST/PORT
my-cluster-kafka-0           my-cluster-kafka-0-amqstr.192.168.64.96.nip.io
my-cluster-kafka-bootstrap    my-cluster-kafka-bootstrap-amqstr.192.168.64.96.nip.io

In this specific environment, the PEM file must have the following extensions:

$ openssl x509 -inform pem -in server.pem -noout -text
# ...
X509v3 extensions:
  X509v3 Basic Constraints: critical
    CA:FALSE
  X509v3 Key Usage:
    Digital Signature, Key Encipherment
  X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication
  X509v3 Subject Alternative Name:
    DNS:my-cluster-kafka-bootstrap-amqstr.192.168.64.96.nip.io, DNS:my-cluster-kafka-0-amqstr.192.168.64.96.nip.io

Once ready, we can create/update the Secret that will host our custom certificate:

$ cat server.pem intermca.pem rootca.pem > fullchain.pem
$ oc create secret generic listener-cert \
    --from-file=server-prk.pem --from-file=fullchain.pem \
    --dry-run -o yaml | oc replace --force -f -

Finally, we just need to configure the external listener by editing the cluster definition and waiting for the rolling update to complete:

$ oc edit kafka $CLUSTER
spec:
  kafka:
    # ...
    listeners:
      plain: {}
      external:
        type: route
        configuration:
          brokerCertChainAndKey:
              secretName: listener-cert
              certificate: fullchain.pem
              key: server-prk.pem

Java client setup

Create the truststore in Java KeyStore (JKS) format in order to verify the identity of the Kafka broker (one-way TLS authentication). Clients only need to trust the root CA public key, regardless of the depth of the chain of trust:

$ keytool -import -noprompt -trustcacerts -alias rootca -file rootca.pem -keystore client-ts.jks -storepass secret

To access Kafka from outside OpenShift, you also need to use this bootstrap URL:

$ echo $(oc get routes $CLUSTER-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'):443

Additional notes

Remember that custom certificates are not managed by the Cluster Operator, so you will have to manually update the OpenShift Secret and clients' truststores during the renewal process. If you update a Kafka listener certificate in a Secret that is already used by a TLS or external listener, a cluster rolling update is also started.

Last updated: March 29, 2023