Kafka Monthly Digest: August 2024

This 79th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in August 2024.

For last month’s digest, see Kafka Monthly Digest: July 2024.

Releases

There is one release in progress:

3.9.0

The release process of 3.9.0 continued. Feature freeze happened on August 8 followed by code freeze on August 30. We are now in the stabilization period which lasts 2 to 3 weeks before publishing the first release candidate. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 11 KIPs (KIP-1075 to KIP-1086, KIP-1084 was skipped). I'll highlight a few of them:

• KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async: With the introduction of Tiered Storage, when handling a LIST_OFFSETS request a broker now may need to read indexes from the remote storage. At the moment LIST_OFFSETS requests are handled synchronously so this may block the request handling thread. This KIP proposes processing LIST_OFFSETS requests asynchronously following the mechanism used by FETCH and PRODUCE requests already.
   
• KIP-1076: Metrics for client applications KIP-714 extension: In 3.7.0, Kafka introduced a mechanism to send Kafka client metrics to brokers (KIP-714). This eases client-side monitoring but only captures metrics from the Kafka clients and ignores the applications embedding them. This KIP builds on this mechanism and aims at providing APIs to allow applications to register their own metrics too so they are captured alongside the Kafka clients metrics.
   
• KIP-1081: Graduation Steps for KIPs: For the past few releases, Kafka has introduced new features marked as "early access", for example Tiered Storage in 3.6.0. However the project never clearly defined what "early access" really means for end users. The goal of this KIP is to introduce graduation steps for complex new features with clear definitions, so users wanting to try features not marked as production-ready know what to expect.

Community Releases

• kroxylicious 0.8.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. The 0.8 release enables hostname verification by default when using TLS. Kroxylicious 0.7, also released this month, added record validation. You can now ensure records sent to a specific topic match a configured schema, otherwise they are rejected.
   
• strimzi-kafka-operator 0.43: Strimzi is a Kubernetes Operator for running Kafka. This new release adds support for Kafka 3.8.0. You can now mount custom volumes in Strimzi resources, for example to store logs, or provide custom Kafka plugins.

Blogs

I selected some interesting blog articles that were published last month:

• Naming Kafka objects (I) – Topics
• Async Processing: The Biggest Upgrade for Kafka Streams Since 0.11
• The hidden costs and risks of implementing Kafka for your enterprise
• The Mysteries of KIP-813

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.