Thanks to changes in Apache Kafka 2.4.0, consumers are no longer required to connect to a leader replica to consume messages. In this article, I introduce you to Apache Kafka’s new
ReplicaSelector interface and its customizable
RackAwareReplicaSelector. I’ll briefly explain the benefits of the new rack-aware selector, then show you how to use it to more efficiently balance load across Amazon Web Services (AWS) availability zones.
For this example, we’ll use Red Hat AMQ Streams with Red Hat OpenShift Container Platform 4.3, running on Amazon AWS.
Continue reading “Consuming messages from closest replicas in Apache Kafka 2.4.0 and AMQ Streams”
As anticipated in the “Additional notes” section of my previous article, starting from Red Hat AMQ Streams 1.4, it is finally possible to use your own custom certificate for encrypting communication between Kafka clients and brokers—without the requirement to provide a CA certificate. The auto-generated and -managed internal CAs will still remain, but only to protect inter-cluster communication.
The user-provided certificate can be used with all listeners that have TLS encryption enabled, such as the route, load balancer, ingress, and NodePort types. In this complete example, we will enable an external route listener for one-way TLS authentication.
You need to have the following in place before you can proceed:
Continue reading “Set up Red Hat AMQ Streams custom certificates on OpenShift (update)”
Kafka Connect is an integration framework that is part of the Apache Kafka project. On Kubernetes and Red Hat OpenShift, you can deploy Kafka Connect using the Strimzi and Red Hat AMQ Streams Operators. Kafka Connect lets users run sink and source connectors. Source connectors are used to load data from an external system into Kafka. Sink connectors work the other way around and let you load data from Kafka into another external system. In most cases, the connectors need to authenticate when connecting to the other systems, so you will need to provide credentials as part of the connector’s configuration. This article shows you how you can use Kubernetes secrets to store the credentials and then use them in the connector’s configuration.
Continue reading “Using secrets in Kafka Connect configuration”