Kafka Monthly Digest: September 2023

October 2, 2023
Mickael Maison

This 68th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in September 2023.

For last month’s digest, see Kafka Monthly Digest: August 2023.

Releases

There is one release in progress: 3.6.0.

3.6.0

The release process for 3.6.0 continued. Satish Duggana published the first release candidate on September 17. During testing, an issue with Kafka Connect and an issue with the ZooKeeper to KRaft migration system tests were found. Satish published RC1 on September 20, unfortunately snappy-java, one of the dependencies of Kafka, announced a high severity CVE on September 24. Satish published RC2 on September 29 and the vote is still on-going. You can find all the details in the release plan.

Kafka Improvement Proposals

Last month, the community submitted 9 KIPs (KIP-975 to KIP-983). I'll highlight a few of them:

  • KIP-975: Docker Image for Apache Kafka: This KIP proposes publishing an official Apache Kafka Docker image. As Kafka is often deployed and run in containerized environments this would help users get started with Kafka.
  • KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect: Connect exposes an endpoint to adjust log levels at runtime. However, it only affects the Connect worker you send the request to, so if you want to adjust log levels of a Connect cluster, you need to send a request per worker. This KIP proposes updating this mechanism to allow updating log levels cluster-wide with a single request.
  • KIP-979: Allow independently stop KRaft controllers or brokers: With KRaft, the kafka-server-start.sh script is used to start both regular brokers and controllers. The kafka-server-stop.sh script has not been updated to handle that, so when running a broker and controller on the same host, the script currently stops everything. This KIP proposes updating the script to select what to stop.
  • KIP-981: Manage Connect topics with custom implementation of Admin: In Kafka 3.4, KIP-787 added the ability to use a custom Admin interface with MirrorMaker. This KIP extends this concept to Kafka Connect. Having a custom Admin interface is useful in environments using a centralized or federated system to manage topics, for example to control capacity or billing.

Community Releases

  • strimzi-kafka-operator 0.37: Strimzi is a Kubernetes Operator for running Kafka. This version requires at least Kubernetes 1.21 to run. Connect and MirrorMaker now use stable identities by default and this new version adds support for the ppc64le platform. There are also important changes around Cruise Control, connector restart behavior and support for OpenTracing has been removed (only OpenTelemetry-based tracing is supported now).

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.