Kafka Monthly Digest: March 2024

This 74th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in March 2024.

For last month’s digest, see Kafka Monthly Digest: February 2024.

Releases

There are two releases in progress, 3.6.2 and 3.8.0:

3.6.2

On March 13, Manikumar Reddy volunteered to be the release manager for Kafka 3.6.2. Since 3.6.1 about 30 issues have been fixed including several blocker bugs. On March 21, he published the first release candidate RC1, however the community quickly identified a couple of issues with it. He published RC2 on March 28 and the vote is currently on-going. You can find the release plan in the wiki.

3.8.0

The work towards 3.8.0 is still ongoing. The next milestone is KIP freeze and it is currently planned for mid May. The release plan is available in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 8 KIPs (KIP-1025 to KIP-1033, KIP-1029 was skipped). I'll highlight a few of them:

• KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header: Since Kafka 3.1, the built-in OAuth authenticator support OIDC. According to the RFC, clients should URL encode their client ID and secret when communicating with the OIDC provider. This is something Kafka does not do at the moment. While several providers don't enforce this requirement, this KIP proposes to optionally opt-in to follow the RFC by introducing a new configuration named sasl.oauthbearer.header.urlencode.
• KIP-1030: Change constraints and default values for various configurations: As Kafka 4.0 approaches, it offers an opportunity for the community to adjust the default values for some configurations. For example, segment.ms, which control how often brokers should roll new segments, can currently be set as low as 1 millisecond. So far the KIP is only considering broker configurations but the discussion is still ongoing to identify configurations that would benefit from having better default values or constraints.
• KIP-1031: Control offset translation in MirrorSourceConnector: In order to translate committed offsets between clusters, MirrorMaker relies on the MirrorSourceConnector to first emit offset-syncs that map offsets between the clusters, and then MirrorCheckpointConnector to use the offset-syncs and commit offsets in the target cluster. Users not interested in translating committed offsets don't run MirrorCheckpointConnector but there is currently no way to prevent MirrorSourceConnector from emitting offset-syncs. This KIP aims at addressing this limitation by introducing a new configuration, emit.offset-syncs.enabled, on MirrorSourceConnector.

Community Releases

• strimzi-kafka-operator 0.40: Strimzi is a Kubernetes Operator for running Kafka. This new release adds support for Kafka 3.7.0. KRaft support is now enabled by default (it used to require setting a feature gate) so administrators can now just choose whether to use KRaft or ZooKeeper when deploying a new cluster. Strimzi now also allows migrating existing ZooKeeper-based clusters to KRaft.
• kroxylicious 0.5.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. In this new release the EnvelopeEncryption filter has been renamed to RecordEncryption to better reflect its role. Kroxylicious now supports Kafka 3.7.0 and brings a number of fixes and improvements especially around TLS and Vault configuration.

Blogs

I selected some interesting blog articles that were published last month:

• Unlocking Kafka's Potential: Tackling Tail Latency with eBPF
• Kafka tiered storage deep dive
• From ZooKeeper to KRaft: How the Kafka migration works
• Mastering Stream Processing - Testing Kafka Streams Windowed Applications

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.