This 68th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in September 2023.
For last month’s digest, see Kafka Monthly Digest: August 2023.
There is one release in progress: 3.6.0.
The release process for 3.6.0 continued. Satish Duggana published the first release candidate on September 17. During testing, an issue with Kafka Connect and an issue with the ZooKeeper to KRaft migration system tests were found. Satish published RC1 on September 20, unfortunately snappy-java, one of the dependencies of Kafka, announced a high severity CVE on September 24. Satish published RC2 on September 29 and the vote is still on-going. You can find all the details in the release plan.
Kafka Improvement Proposals
Last month, the community submitted 9 KIPs (KIP-975 to KIP-983). I'll highlight a few of them:
- KIP-975: Docker Image for Apache Kafka: This KIP proposes publishing an official Apache Kafka Docker image. As Kafka is often deployed and run in containerized environments this would help users get started with Kafka.
- KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect: Connect exposes an endpoint to adjust log levels at runtime. However, it only affects the Connect worker you send the request to, so if you want to adjust log levels of a Connect cluster, you need to send a request per worker. This KIP proposes updating this mechanism to allow updating log levels cluster-wide with a single request.
- KIP-979: Allow independently stop KRaft controllers or brokers: With KRaft, the
kafka-server-start.shscript is used to start both regular brokers and controllers. The
kafka-server-stop.shscript has not been updated to handle that, so when running a broker and controller on the same host, the script currently stops everything. This KIP proposes updating the script to select what to stop.
- KIP-981: Manage Connect topics with custom implementation of Admin: In Kafka 3.4, KIP-787 added the ability to use a custom Admin interface with MirrorMaker. This KIP extends this concept to Kafka Connect. Having a custom Admin interface is useful in environments using a centralized or federated system to manage topics, for example to control capacity or billing.
- strimzi-kafka-operator 0.37: Strimzi is a Kubernetes Operator for running Kafka. This version requires at least Kubernetes 1.21 to run. Connect and MirrorMaker now use stable identities by default and this new version adds support for the ppc64le platform. There are also important changes around Cruise Control, connector restart behavior and support for OpenTracing has been removed (only OpenTelemetry-based tracing is supported now).
I selected some interesting blog articles that were published last month:
- Scaling Kafka to Support PayPal’s Data Growth
- Apache Kafka Message Compression
- Online machine learning with the data streams from the database
- The Stairway to Apache Kafka® Tiered Storage
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.