Streams for Apache Kafka: Quick setup from the OpenShift console

Follow the steps below to quickly get a basic Apache Kafka cluster up and running in Red Hat OpenShift with the operator for streams for Apache Kafka. You will need to access a Red Hat OpenShift Container Platform cluster using an account with cluster-admin permissions.

hero

Installing AMQ Streams Operator from OperatorHub

  1. Launch your web browser and open a new tab with the Administrator view of the web console for your OpenShift cluster. Log in to the console as an administrator.
  2. After logging in, click OperatorsOperatorHub in the left side navigation menu.
  3. On the Project drop-down menu at the top of the OperatorHub page, select the project in which you want to install the operator.
  4. There are quite a few different operators shown on the page. You can use the Filter by keyword box at the top of the page to quickly locate the AMQ Streams Operator. Just input the text “AMQ Streams” and click on the result operator, as shown in Figure 1.

    Locating the AMQ Streams Operator in OperatorHub

    Figure 1: Locating the AMQ Streams Operator in OperatorHub.

  5. Click the AMQ Streams operator. In the dialog box that appears, click Install as illustrated in Figure 2.

    Installing the AMQ Streams Operator.

    Figure 2: Installing the AMQ Streams Operator.

    It will bring you to the AMQ Streams Operator configuration page, as illustrated in Figure 3.

    The AMQ Streams Operator configuration page

    Figure 3: The AMQ Streams Operator configuration page.

    The operator supports watching just one namespace or the whole cluster. For this quick setup, we will use the default option and have the operator watch all namespaces.

  6. Click Install. When the operator installation is complete, it will show you that the operator is ready to use, as illustrated in Figure 4.

    The installed AMQ Streams operator once ready for use

    Figure 4: The installed AMQ Streams operator once ready for use.

  7. Click View Operator, and you will get the installed AMQ Streams Operator details page.

Creating Apache Kafka® clusters with the AMQ Streams operator

  1. On the Project drop-down menu at the top of the page, select the project for which you want to create the cluster.

    If there is no project already created, you can create a new one as illustrated in Figure 5.

    The Create Project window.

    Figure 5: The Create Project window.

    Click on the operator name to get back to the details page.

  2. On the operator details page, click Create instance under the Kafka box as illustrated in Figure 6. 

    The API options, each with an option to create a new instance

    Figure 6: The API options, each with an option to create a new instance.

  3. Click on YAML View and paste the following manifest, as shown in Figure 7.

    apiVersion: kafka.strimzi.io/v1beta2
                    kind: Kafka
                    metadata: 
                      name: my-cluster
                    spec:
                      kafka:
                        replicas: 3
                        listeners:
                          - name: external
                            port: 9092
                            type: route
                            tls: true
                        storage:
                          type: ephemeral
                      zookeeper:
                        replicas: 3
                        storage:
                          type: ephemeral
                      entityOperator:
                        topicOperator: {}

    Defining the YAML for the new project

    Figure 7: Defining the YAML for the new project.

    On the creation page, you can change or modify the basic template. For this quickstart, we will go with the defaults. This will deploy a 3-node Kafka cluster with 3 nodes ZooKeeper and enable one external listener for the cluster using ephemeral storage.

  4. Click Create at the bottom of the page.
  5. The console will show you the created Kafka instance and its status. Click on the Name of the Kafka instance to see more details. 

    Viewing the newly created Kafka instance on the Operator details page

    Figure 8: Viewing the newly created Kafka instance on the Operator details page.

    If you scroll down, the details page shows more information about the instance, as illustrated in Figure 9.

    Details about the Kafka instance you just created.

    Figure 9: Details about the Kafka instance you just created.

This confirms you have successfully created one Apache Kafka cluster using AMQ Streams in Red Hat OpenShift.

Start sending and receiving from a Topic

Test using an external Red Hat build of Apache Camel application. You will need to have Maven installed and a Java 8 JDK.

  1. Clone this Git repo to test access to your new Kafka cluster:

    $ git clone https://github.com/hguerrero/amq-examples.git    
  2. Switch to the camel-kafka-demo folder:

    $ cd amq-examples/camel-kafka-demo/
  3. Log in to the OpenShift cluster with cluster-admin privileges, for example:

    $ oc login -u system:admin
  4. Make sure you are in the correct project namespace:

    $ oc project kafka
  5. As we are using Routes for external access to the cluster, we need the cluster CA certificate to enable TLS in the client. Extract the public certificate of the broker certification authority:

    $ oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > src/main/resources/ca.crt       
  6. Import the trusted cert to a Keystore:

    $ keytool -import -trustcacerts -alias root -file src/main/resources/ca.crt -keystore src/main/resources/keystore.jks -storepass password -noprompt     

    Now you can run the Apache Camel application to send and receive messages to the Kafka cluster using the following Maven command (This step currently requires Java 8):

    $ BOOTSTRAP_SERVER="`oc get routes my-cluster-kafka-external-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'`:443" mvn clean package spring-boot:run

    After finishing the clean and package phases, you will see the Spring Boot application start creating a producer and consumer, sending and receiving messages from the my-topic Kafka topic.

    14:36:18.170 [main] INFO  com.redhat.kafkademo.Application - Started Application in 12.051 seconds (JVM running for 12.917)
                    14:36:18.490 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  o.a.k.c.c.me.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=6de87ffa-c7cf-441b-b1f8-e55daabc8d12] Discovered coordinator my-cluster-kafka-1-myproject.192.168.99.100.nip.io:443 (id: 2147483646 rack: null)
                    14:36:18.498 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  o.a.k.c.c.me.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=6de87ffa-c7cf-441b-b1f8-e55daabc8d12] Revoking previously assigned partitions []
                    14:36:18.498 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  o.a.k.c.c.me.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=6de87ffa-c7cf-441b-b1f8-e55daabc8d12] (Re-)joining group
                    14:36:19.070 [Camel (MyCamel) thread #3 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-2
                    14:36:19.987 [Camel (MyCamel) thread #4 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-4
                    14:36:20.982 [Camel (MyCamel) thread #5 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-6
                    14:36:21.620 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  o.a.k.c.c.me.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=6de87ffa-c7cf-441b-b1f8-e55daabc8d12] Successfully joined group with generation 1
                    14:36:21.621 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  o.a.k.c.c.me.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=6de87ffa-c7cf-441b-b1f8-e55daabc8d12] Setting newly assigned partitions [my-topic-0, my-topic-1, my-topic-2]
                    14:36:21.959 [Camel (MyCamel) thread #6 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-8
                    14:36:21.973 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  consumer-route - consumer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-8
                    14:36:22.970 [Camel (MyCamel) thread #7 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-11
                    14:36:22.975 [Camel (MyCamel) thread #1 - KafkaConsumer[my-topic]] INFO  consumer-route - consumer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-11
                    14:36:23.968 [Camel (MyCamel) thread #8 - KafkaProducer[my-topic]] INFO  producer-route - producer >>> Hello World from camel-context.xml with ID ID-hguerrer-osx-1540578972584-0-14  
  7. You're done! Press Ctrl + C to stop running the program.

Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.