This 53rd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in June 2022.
For last month’s digest, see Kafka Monthly Digest: May 2022.
There is currently one release in progress: 3.3.0.
The release process for 3.3.0 continued. KIP freeze happened on June 15, and feature freeze happened on July 6. You can find the release plan in the wiki.
Kafka Improvement Proposals
Last month, the community submitted 4 KIPs (KIP-847 to KIP-850). I'll highlight a few of them:
KIP-847: Add ProducerCount metrics: This KIP proposes adding new metrics to track how many idempotent and transactional producers are connected to each broker. For each idempotent or transactional producer connected, brokers keep in memory some metadata about it. The goal is to improve observability and help diagnose whether there may be too many producers connected.
KIP-848: The Next Generation of the Consumer Rebalance Protocol: The Kafka group membership protocol allows distributing resources amongst client instances. It is used by consumers when they are in groups, but also by Connect and Streams. While this protocol is very flexible, over the years a few pain points have been identified. This KIP proposes significantly updating this protocol to make rebalances lightweight and simplify troubleshooting by moving most of the logic to the server side.
KIP-849: Expose logdirs total and usable space via kafka-log-dirs.sh: In Kafka 3.2, KIP-827 exposed the size of usable and total space of log directories via the Admin API. This KIP aims at updating
kafka-log-dirs.shto include these values in its output so administrators can easily access them.
Librdkafka 1.9.0. Librdkafka is a Kafka client in C/C++. This new release adds support for OAuthbearer OIDC and admin APIs for managing ACLs. As always, it also brings in many improvements and bug fixes.
eachMessagehandlers. This release also contains some important bug fixes.
I selected some interesting blog posts and articles that were published last month:
- High watermark: Distributed design patterns
- Rack awareness in Kafka Streams
- Debezium to Snowflake: Lessons learned building data replication in production
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.Last updated: September 20, 2023