Kafka Monthly Digest: May 2024

This 76th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in May 2024.

For last month’s digest, see Kafka Monthly Digest: April 2024.

Releases

There are two releases in progress, 3.8.0 and 3.7.1:

3.8.0

The work towards 3.8.0 is still ongoing. KIP freeze happened on May 15 and Josep Prat cut the new 3.8 branch with feature freeze on May 29. Unfortunately work on KIP-853 and KIP-966 is not complete yet, but the community agreed to merge them pass the feature freeze because Kafka 3.8 is at the moment expected to be the last 3.X release. The next milestone is code freeze on June 12. The release plan is available in the wiki.

3.7.1

There are still a few blocker issues targeting 3.7.1. Once these are resolved, Igor Soarez will publish the first release candidate. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 9 KIPs (KIP-1041 to KIP-1049). I'll highlight a few of them:

• KIP-1042: Support for wildcard when creating new acls: When creating ACLs in Kafka, the resources they apply to can be defined literally (for example the exact topic names), or via a prefix to apply to all resources that start with it. This KIP aims at providing full support for wildcards so resources can be defined more freely.
• KIP-1045: Move MockAdminClient to public api: Kafka offers the MockProducer and MockConsumer classes to help users test their applications. There's also a MockAdminClient class but at the moment it's not part of the public API and is only used by Kafka's own test suites. This KIP proposes moving this MockAdminClient class to the public API so users can also use it to test their applications.
• KIP-1046: Expose producer.id.expiration.check.interval.ms as dynamic broker configuration: Administrators can use the producer.id.expiration.ms broker configuration to define when inactive producer IDs can be deleted. However internally brokers only check whether producer IDs need to be deleted every 10 minutes, meaning that setting producer.id.expiration.ms to 10000 or 50000 ends up having the same behavior. This KIP proposes exposing a new configuration, producer.id.expiration.check.interval.ms, to allow defining how often brokers check for expired producer IDs.

Community Releases

• strimzi-kafka-operator 0.41: Strimzi is a Kubernetes Operator for running Apache Kafka. This release adds support for Kafka 3.6.2 and support for JBOD in KRaft mode. The Topic operator now allows changing the replication factor of topics, and there are new metrics to monitor the expiration of certificates.
• librdkafka 2.4: Librdkafka is an Apache Kafka client in C/C++. This release adds early access support for the new consumer group protocol (KIP-848). It also contains improvements around handling producer errors, topic IDs, as well as many bug fixes.
• akhq 0.25: AKHQ is a GUI for Apache Kafka. This new release completely revamps the ACL system to support multiple clusters RBAC. It also contains plenty of improvements around topics, records and schemas management, and dozens of bug fixes.

Blogs

I selected some interesting blog articles that were published last month:

• Preventing and Fixing Bad Data in Event Streams — Part 1
• How We Solve Load Balancing Challenges in Apache Kafka
• Contributing to Apache Kafka®: How to Write a KIP
• What’s in a name? Making Sense of Apache Kafka’s auto.offset.reset

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.