We have pretty exciting news this week as Red Hat is announcing the General Availability of their Apache Kafka Kubernetes operator. Red Hat AMQ Streams delivers the mechanisms for managing Apache Kafka on top of OpenShift, our enterprise distribution for Kubernetes.
Everything started last May 2018 when David Ingham (@dingha) unveiled the Developer Preview as new addition to the Red Hat AMQ offering. Red Hat AMQ Streams focuses on running Apache Kafka on OpenShift. In the microservices world, where several components need to rely on a high throughput communication mechanism, Apache Kafka has made a name for itself for being a leading real-time, distributed messaging platform for building data pipelines and streaming applications.
A leader in the traditional infrastructure deployments, Apache Kafka was missing pieces to be a easily usable container-native citizen in the new Kubernetes era. As a result, a team grouped in 2017 to create the upstream Strimzi project. This team worked to apply the new operator pattern to solve the gaps. With the development of new components deployed along the traditional Apache Kafka broker, these new Kubernetes operators are now able to manage cluster wide resources as well as entities as topics and authentication users.
Even more, these Kubernetes operators are pretty simple to use and cover most of the more common configuration management of a cluster. After doing some Kubernetes Custom Resource Definition installations in OpenShift, any user is able to create an Apache Kafka cluster by just creating a new Kafka resource definition. Then, the cluster operator will take that definition and provision the required components to have a fully deployed cluster on top of your OpenShift infrastructure. The same applies for the creation of Topics and Users for that same Kafka cluster.
Finally, the most interesting part, is the simple configuration to provision a cluster that's easily available from outside the OpenShift cluster via NodePorts, Load Balancers or Routes, the latter being the easiest way to start working. Depending on your application requirements, you will be able to choose from using the simple secure over TLS approach or the traditional NodePort settings. As a result, the AMQ Streams operators embedded logic creates dynamic services, routes and certificates to access the Apache Kafka cluster from external clients.
Yes, it sounds like magic. That’s how good it is! Give Red Hat AMQ Streams a try by downloading Red Hat Container Development Kit for an OpenShift development environment and following our Getting Started Guide. Also, be aware of my HOW-TO guide for running Kafka on OpenShift, the enterprise Kubernetes!
Here are some other articles on Apache Kafka and Red Hat AMQ Streams:
Last updated: June 6, 2023