Using secrets in Kafka Connect configuration

Using secrets in Kafka Connect configuration

Kafka Connect is an integration framework that is part of the Apache Kafka project. On Kubernetes and Red Hat OpenShift, you can deploy Kafka Connect using the Strimzi and Red Hat AMQ Streams Operators. Kafka Connect lets users run sink and source connectors. Source connectors are used to load data from an external system into Kafka. Sink connectors work the other way around and let you load data from Kafka into another external system. In most cases, the connectors need to authenticate when connecting to the other systems, so you will need to provide credentials as part of the connector’s configuration. This article shows you how you can use Kubernetes secrets to store the credentials and then use them in the connector’s configuration.

In this article, I will use an S3 source connector, which is one of the Apache Camel Kafka connectors. To learn more about Apache Camel Kafka connectors, you can start with this blog post. This connector is used just as an example of how to configure a connector to access a secret. You can use this same procedure with any connector, as there is nothing special required from the connector itself. We will use the S3 connector to connect to Amazon AWS S3 storage and load files from an S3 bucket into an Apache Kafka topic. In order to connect to S3 storage, we will need to specify the AWS credentials: the access key and the secret key. So, let’s start by preparing the secret with the credentials.

Creating a secret with the credentials

First, we will create a simple properties file called aws-credentials.properties, which should look like this:

aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The credentials you use in this properties file need to have access to the S3 bucket we will read from. Once we have the properties file with the credentials ready, we have to create the secret from this file. You can use the following command to do so:

$ kubectl create secret generic aws-credentials --from-file=./aws-credentials.properties

Everything you need to grow your career.

With your free Red Hat Developer program membership, unlock our library of cheat sheets and ebooks on next-generation application development.

SIGN UP

Building new container images with the connector

Next, we need to prepare a new Docker image with our connector. When using Strimzi, the Dockerfile for adding the connector should look something like this:

FROM strimzi/kafka:0.16.1-kafka-2.4.0
USER root:root
COPY ./my-plugins/ /opt/kafka/plugins/
USER 1001

When using AMQ Streams, it should look like this:

FROM registry.redhat.io/amq7/amq-streams-kafka-23:1.3.0
USER root:root
COPY ./my-plugins/ /opt/kafka/plugins/
USER jboss:jboss

Use the Dockerfile to build a container image with the connectors you need, and push them into your registry. If you don’t have your own private registry, you can use one of the public registries such as Quay or Docker Hub.

Deploying Apache Kafka Connect

Once we have the container image, we can finally deploy Apache Kafka Connect. You can do this by creating the following custom resource:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  image: docker.io/scholzj/kafka:camel-kafka-2.4.0
  replicas: 3
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  externalConfiguration:
    volumes:
      - name: aws-credentials
        secret:
          secretName: aws-credentials
  config:
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false

Let’s look in more detail at several parts of the custom resource. First of all, notice the image field, which tells the Operator deploying Apache Kafka Connect to use the right image with the added connectors. In my case, I pushed the container image built in previous section to Docker Hub as scholzj/kafka:camel-kafka-2.4.0, so my configuration looks like this:

image: docker.io/scholzj/kafka:camel-kafka-2.4.0

Next, notice the externalConfiguration section:

externalConfiguration:
  volumes:
    - name: aws-credentials
      secret:
        secretName: aws-credentials

In this section, we instruct the Operator to mount the Kubernetes secret aws-credentialsthat we created at the beginning of this article into the Apache Kafka Connect pods. The secrets listed here will be mounted in the path /opt/kafka/external-configuration/<secretName> where the <secretName> is the name of the secret.

And finally, in the config section, we enable the FileConfigProvider as a configuration provider in Apache Kafka Connect:

config:
  config.providers: file
  config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider

Configuration providers are a way of loading configuration values from another source instead of specifying them in the configuration directly. In this case, we create the configuration provider namef file, which will use the FileConfigProvider class. This configuration provider is part of Apache Kafka. FileConfigProvider can read properties files and extract values from them, and we will use it to load API keys for our Amazon AWS account.

Creating the connector using the Apache Kafka Connect REST API

Usually, we have to wait a minute or two for the Apache Kafka Connect deployment to become ready. And once it is ready, we can create the connector instance. In older versions of Strimzi and Red Hat AMQ Streams, you have to do that using the REST API. We can create the connector by posting the following JSON:

{
 "name": "s3-connector",
 "config": {
   "connector.class": "org.apache.camel.kafkaconnector.CamelSourceConnector",
   "tasks.max": "1",
   "camel.source.kafka.topic": "s3-topic",
   "camel.source.maxPollDuration": "10000",
   "camel.source.url": "aws-s3://camel-connector-test?autocloseBody=false",
   "key.converter": "org.apache.kafka.connect.storage.StringConverter",
   "value.converter": "org.apache.camel.kafkaconnector.converters.S3ObjectConverter",
   "camel.component.aws-s3.configuration.access-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}",
   "camel.component.aws-s3.configuration.secret-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}",
   "camel.component.aws-s3.configuration.region": "US_EAST_1"
   }
}

The connector configuration contains the AWS API keys in the fields camel.component.aws-s3.configuration.access-key and camel.component.aws-s3.configuration.secret-key. Instead of using the values directly, we just reference the file configuration provider to load the fields aws_access_key_id and aws_secret_access_key from our aws-credentials.properties file.

Notice how we reference the config provider, tell it the path to the file it should use, and include the name of the key to extract:

"camel.component.aws-s3.configuration.access-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}"

and:

"camel.component.aws-s3.configuration.secret-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}"

You can POST the results to the Apache Kafka Connect REST API, for example, using curl:

$ curl -X POST -H "Content-Type: application/json" -d connector-config.json http://my-connect-cluster-connect-api:8083/connectors

One of the advantages of using the configuration providers is that even when you later get the connector configuration, it will still contain the configuration provider and not the values you want to keep secret:

$ curl http://my-connect-cluster-connect-api:8083/connectors/s3-connector
{
  "name": "s3-connector",
  "config": {
    "connector.class": "org.apache.camel.kafkaconnector.CamelSourceConnector",
    "camel.source.maxPollDuration": "10000",
    "camel.source.url": "aws-s3://camel-connector-test?autocloseBody=false",
    "camel.component.aws-s3.configuration.region": "US_EAST_1",
    "camel.component.aws-s3.configuration.secret-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}",
    "tasks.max": "1",
    "name": "s3-connector",
    "value.converter": "org.apache.camel.kafkaconnector.converters.S3ObjectConverter",
    "camel.component.aws-s3.configuration.access-key": "${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "camel.source.kafka.topic": "s3-topic"
  },
  "tasks": [
    {
      "connector": "s3-connector",
      "task": 0
    }
  ],
  "type": "source"
}

Creating the connector with the Strimzi connector Operator

When using Strimzi 0.16.0 or newer, we can also use the new connector Operator. It lets us create the connector using the following custom resource YAML (you can use the configuration provider directly in the KafkaConnector custom resource as well):

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSourceConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.camel.kafkaconnector.converters.S3ObjectConverter
    camel.source.kafka.topic: s3-topic
    camel.source.url: aws-s3://camel-connector-test?autocloseBody=false
    camel.source.maxPollDuration: 10000
    camel.component.aws-s3.configuration.access-key: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}
    camel.component.aws-s3.configuration.secret-key: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}
    camel.component.aws-s3.configuration.region: US_EAST_1

Conclusion

The security of Kubernetes secrets has its limitations, any user who can exec into the container will be able to read the mounted secrets anyway. This process at least prevents the confidential information, such as credentials or API keys, from being exposed through the REST API or in the KafkaConnector custom resources.

Join Red Hat Developer and get access to handy cheat sheets, free books, and product downloads.

To learn more, visit our Linux containers or microservices pages.

Join Red Hat Developer and get access to handy cheat sheets, free books, and product downloads that can help you with your microservices and container application development.

Share