Kafka Monthly Digest: November 2024

This 82nd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in November 2024.

For last month’s digest, see Kafka Monthly Digest: October 2024.

Releases

There is 1 new release and 2 releases are in progress:

3.9.0

On November 8, Colin McCabe released Apache Kafka 3.9.0. You can find the announcement on the Apache Kafka blog. You can also check the release notes and the release plan in the wiki for the full details.

This is the final minor release in the 3.X line. Kafka 4.0 will no longer support ZooKeeper, so clusters still using ZooKeeper must migrate to KRaft using 3.9 before being able to upgrade to 4.0.

This new minor release brings several new features and many bug fixes.

Kafka brokers and client

Updates to the Kafka broker and clients include the following:

• Tiered storage is now production ready. You can enable and disable it per topic and set network quotas. (KIP-950, KIP-956, KIP-1005, KIP-1057)
• New dynamic KRaft quorums. This allows adding and removing controller nodes at runtime. Until now KRaft quorums required to define all controllers upfront and they could not change. From 3.9 the recommended deployment method is to use a dynamic quorum. However note that Kafka currently does not provide a way to update existing static quorums to be dynamic, so only new quorums deployed with Kafka 3.9 can be dynamic. (KIP-853)
• New configuration, sasl.oauthbearer.header.urlencode, to make the OAuth authentication with an OIDC provider compliant with RFC-6749. (KIP-1024) 

Kafka Connect

Updates to Kafka Connect include the following:

• New REST endpoint, GET /health to check the state of a Connect worker. (KIP-1017)
• Consumer offset translation can now be fully disabled by setting emit.offset-syncs.enabled=false in MirrorSourceConnector's configuration. (KIP-1031)
• The InsertField, ExtractField, HeaderFrom, Cast, SetSchemaMetadata, TimestampConverter and MaskField transformations now support the replace.null.with.default configuration (like JsonConverter) to better handle null values. (KIP-1040)

Kafka Streams

• New ProcessingExceptionHandler interface to simplify the handling of processing errors. (KIP-1033)
• A new configuration, log.summary.interval.ms, allows setting how often, or even disable if set to 0, the summary information is logged. (KIP-1049)
   

4.0.0

Progress on Kafka 4.0.0 continued. KIP freeze happened on November 20. The next milestone is feature freeze on December 11. You can find the release plan in the wiki.

3.7.2

All the fixes we wanted to backport for Kafka 3.7.2 have been merged. We should expect it to release in December. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 13 KIPs (KIP-1106 to KIP-1118). I'll highlight a few of them:

• KIP-1109: Unifying Kafka Consumer Topic Metrics: For historical reasons, dots in topic names are replaced by underscores in the Kafka Consumer metrics. This can lead to confusion and it is not consistent with other metrics, for example from the Producer, which preserve topic names. This KIP aims at fixing this inconsistency by keeping topic names unchanged in the Consumer metrics.
   
• KIP-1111: Enforcing Explicit Naming for Kafka Streams Internal Topics: Kafka Streams creates internal topics to store state. The names of these topics are computed when the Streams application starts, and while they follow a pattern it's not always easy to guess what the actual names will be for complex topologies. This KIP proposes a mechanism so users can explicitly provide the names of all internal topics used by a Kafka Streams application.
   
• KIP-1117: Support keystore with multiple alias entries: When a keystore is provided as part of the TLS configuration, Kafka always picks the first key. This KIP proposes introducing a new configuration, ssl.keystore.alias, to explicitly select the key to use in a keystore.

Community Releases

• node-rdkafka 3.2. Node-rdkafka is a Node.js Kafka client based on librdkafka. This new release is now based on librdkafka 2.6.0 and adds support for cooperative assignment in the Consumer.

Blogs

I selected some interesting blog articles that were published last month:

• Auto-rebalancing on cluster scaling
• Dynamic Kafka controller quorum
• Deep Dive into Handling Consumer Fetch Requests: Kafka Producer and Consumer Internals, Part 4

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.