Kafka Monthly Digest: September 2024

This 80th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in September 2024.

For last month’s digest, see Kafka Monthly Digest: August 2024.

Releases

There are 3 releases in progress:

3.9.0

The release process of 3.9.0 continued. The stabilization period ended mid September and Colin McCabe published the first release candidate on September 18. The community found a few issues including bugs in the new KRaft controller membership management feature (KAFKA-17608, KAFKA-17604). These have all been fixed now so we should get a new release candidate soon. You can find the release plan in the wiki.

3.8.1

On September 20 Josep Prat volunteered to be the release manager for Kafka 3.8.1. Since 3.8.0, 10 issues have been fixed. The release is planned for October. You can find the release plan in the wiki.

4.0.0

On September 23 David Jacot volunteered to be the release manager for Kafka 4.0.0. Kafka 4.0.0 is the next major release and will completely remove ZooKeeper. The release plan is still being worked on.

Kafka Improvement Proposals

Last month, the community submitted 9 KIPs (KIP-1087 to KIP-1095). I'll highlight a few of them:
 

• KIP-1089: Allow disabling heartbeats replication in MirrorSourceConnector: At the moment MirrorSourceConnector always mirrors heartbeats topics. This can be an issue if users have topics that match the heartbeats topic detection logic as these will be unconditionally mirrored. This KIP introduces a new configuration heartbeats.replication.enabled to change this behavior.
   
• KIP-1091: Improved Kafka Streams operator metrics: Kafka Streams already exposes many fine grained metrics, but this KIP proposes adding a few high level metrics to quickly gauge the health of an application such as the state of the Streams instance and all its threads.
   
• KIP-1092: Extend Consumer#close with an option to leave the group or not: When a consumer instance shuts down, it does not necessarily sends a LeaveGroup request to leave its consumer group. This can happen for various reasons, for example a consumer with group.instance.id set or in Kafka Streams. This KIP proposes exposing this option directly in the consumer API, so users can choose whether a consumer shutting down should explicitly leave its group and trigger a rebalance, or instead remain in the group if it's expected to restart quickly otherwise the coordinator will trigger a rebalance once the session expires.
   
• KIP-1093: Add more observability for MirrorMaker2: This KIP aims at adding extra metrics to MirrorMaker to further improve its observability. It proposes adding metrics tracking the number of topics, partitions and groups mirrored, and information (execution times, failures) about the internal jobs the connectors run.

Community Releases

None of the projects I follow have made a release in September. If you know an open source community project that I don't cover, please let me know on X or Mastodon.

Blogs

I selected some interesting blog articles that were published last month:

• Kafka protocol practical guide
• Naming Kafka objects (II) – Producers and Consumers
   

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.