Kafka Monthly Digest: April 2025

May 6, 2025
Mickael Maison
Related topics:
Kafka
Related products:
Streams for Apache Kafka

This 87th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in April 2025.

For last month’s digest, see Kafka Monthly Digest: March 2025.

Releases

There are 2 releases in progress:

3.9.1

The release process of Kafka 3.9.1 continued. TengYao Chi published the first release candidate, 3.9.1 RC0, on April 16. Unfortunately an issue in the release process was spotted. The next RC, 3.9.1 RC1, was then published on April 23 and the vote is currently on-going. You can find the release plan in the wiki.

4.1.0

The next milestone for Kafka 4.1.0 is KIP freeze on May 14. The release is planned for July. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 22 KIPs (KIP-1152 to KIP-1175, 1167, 1168 and 1169 were skipped. Also KIP-1150 which was skipped last month has been created). I'll highlight a few of them:

First we have 2 KIPs that propose introducing topics without inter-broker replication. The reason is that in many cases inter-broker traffic is the biggest cost when running a Kafka cluster. In both proposals, topic data would be stored in remote storage (typically cloud object storage) and followers would retrieve records from that place instead of directly replicating from the leaders. However the KIPs propose different approaches:

  • KIP-1150: Diskless Topics: As the title hints, the goal is to introduce topics whose data won't be stored on brokers but directly sent to cloud object storage. To optimize writes to the remote storage system, brokers can group records from different topic-partitions into an object called a shared log segment object. A new coordination layer is then used to retrieve specific records from these objects. This proposal comes with a few follow up KIPs: KIP-1163: Diskless Core, KIP-1164: Topic Based Batch Coordinator and KIP-1165: Object Compaction for Diskless.
     
  • KIP-1176: Tiered Storage for Active Log Segment: This second approach extends concepts from tiered storage (KIP-405) and proposes uploading the active segment to remote storage. In this case, followers would still send Fetch requests to leaders, but they will only be used to update high watermarks. Leaders will not send records back, the followers will have to fetch them from the remote storage.

Other KIPs submitted this month:

  • KIP-1155: Metadata Version Downgrades: The cluster metadata is managed by the KRaft controllers. It is versioned and effectively describes the features available in a Kafka cluster. KIP-778 defined the mechanism to update the metadata version (via the updateFeatures() API or the kafka-features.sh tool) which is necessary when upgrading the Kafka version. This KIP introduces a process to downgrade the metadata version. This could be useful in case a specific metadata version has a bug, or if a Kafka release has an issue and users need to revert to an older release after upgrading.
     
  • KIP-1156: Define an official backward compatibility contract: Historically the Apache Kafka project has maintained backwards compatibility with all previous releases. A 0.9 client released almost 10 years ago can still connect to a cluster running 3.9. However this comes at a significant cost for the project and as of Kafka 4.0, backwards compatibility is now only maintained from Kafka 2.1. This KIP aims at defining a backward compatibility contract so it's clear to users when a specific release is expected to not be compatible anymore.
     
  • KIP-1159: Large message reference based Serializer: Kafka uses the message.max.bytes configuration to define the largest message size it can handle. It defaults to 1MiB and it's recommended to keep it within that order of magnitude. This KIP proposes introducing a large message SerDes, using reference-based messaging, to enable use cases that require larger messages.

Community Releases

  • Debezium 3.1: Debezium is a Change Data Capture platform. This new release is packed with new features and improvements for pretty much all connectors (JDBC, MariaDB, MySQL, Oracle, SQL Server, Vitess).
     
  • Librdkafka 2.10. Librdkafka is a Kafka client in C/C++. This release improves support for KIP-848 (new consumer protocol), brings many improvements around handling of cluster metadata and fixes dozens of general and consumer client bugs.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.