Kafka Monthly Digest: April 2024

May 2, 2024
Mickael Maison

This 75th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in April 2024.

For last month’s digest, see Kafka Monthly Digest: March 2024.

Releases

There is one new release, 3.6.2, and two releases in progress, 3.8.0 and 3.7.1:

3.6.2

On April 4, Manikumar Reddy released Kafka 3.6.2. This bugfix release contains 28 fixes since 3.6.1, including a handful of blocker issues. For more details, you can check the release announcement on the blog and the release notes.

3.8.0

The work towards 3.8.0 is still ongoing. The next milestone is KIP freeze and it is planned for mid May. The release plan is available in the wiki.

3.7.1

On April 25, Igor Soarez volunteered to run the 3.7.1 release. The release is currently targeted for May/June and you can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 7 KIPs (KIP-1034 to KIP-1040). I'll highlight a few of them:

  • KIP-1034: Dead letter queue in Kafka Streams: When Kafka Streams encounters an unprocessable record, it basically offers two options. The record can either be ignored and skipped, or the Kafka Streams application crashes. Dead Letter Queues (DLQ) are a common mechanism to improve this scenario as they allow storing unprocessable records in a separate place, in this case a topic, allowing the application to carry on and ensuring these records are not lost. For example, Kafka Connect supports dead letter queues in sink pipelines. This KIP's goal is to bring a similar mechanism to Kafka Streams.
  • KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission: To investigate and resolve issues with hanging transactions, tooling was added to the Admin client via KIP-664. This includes listing and describing the state of transactions, as well as aborting transactions. In order to abort transactions, the Admin client use the WriteTxnMarker API. This API requires the ClusterAction ACL on the Cluster resource. As most of the other administrative actions the Admin client performs require the Alter ACL on the Cluster resource, this KIP proposes allowing WriteTxnMarker if the Admin client has either ClusterAction or Alter on the Cluster resource, so the required ACLs for the various administrative actions are more consistent.
  • KIP-1039: Disable automatic topic creation for MirrorMaker2 consumers: By default, when a consumer subscribes to a topic, if that topic does not exist it gets automatically created. This behavior is specified by the allow.auto.create.topics configuration which defaults to true. With MirrorMaker this can lead to a confusing behavior when a topic being mirrored is deleted from the source cluster. As MirrorMaker only refreshes topics periodically, most of the time there's a short window when MirrorMaker is still running consumers after a topic has been deleted, and this can trigger the topic to be re-created. Users can explicitly disable allow.auto.create.topics but this KIP proposes disabling this setting by default in MirrorMaker.

Community Releases

  • Debezium 2.6: Debezium 2.6 introduces a new connector for Db2 for iSeries. It also unifies the snapshot modes across all the connectors. Another new feature is the asynchronous Embedded Engine. Debezium can be used as a library and embedded in applications, as it's done in Apache Flink CDC for example. This runtime mode is called Embedded Engine. The new asynchronous version of the Embedded Engine aims at improving performance and resource utilization in pipelines that can be parallelized by allowing for example to run multiple tasks for a connector, and run transformations and serialization on dedicated threads. There are also a few breaking changes that are worth checking before you upgrade.
  • node-rdkafka 3.0: This new release of node-rdkafka drops support for Node.js < 16 and adds support for Node.js 21. It now also supports OAuthBearer authentication.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.