Kafka Monthly Digest: March 2025

April 1, 2025
Mickael Maison
Related topics:
Kafka
Related products:
Streams for Apache Kafka

This 86th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in March 2025.

For last month’s digest, see Kafka Monthly Digest: February 2025
 

Releases

There is 1 new release, and 2 releases are in progress:

4.0.0

On March 18, after 4 release candidates, David Jacot published Apache Kafka 4.0.0. You can find the announcement on the Apache Kafka blog. You can also check the release notes and the release plan in the wiki.

Kafka 4.0.0 is a major release introducing significant general changes:

  • It removes ZooKeeper support, so clusters still using ZooKeeper must first migrate to KRaft using 3.9 before being able to upgrade to 4.0. As with any major release, it's recommended to check the upgrade guide and the notable changes sections in the documentation before upgrading.
  • It drops support for Java 8. Clients and Streams now require Java 11, while brokers, tools and Connect now require Java 17 (KIP-750 and KIP-1013).
  • It drops support for old API protocols, the new baseline is Kafka 2.1. Java clients older than 2.1 won't work with 4.0.0 brokers (KIP-896), similarly 4.0 clients won't work with brokers older than 2.1. The old message formats v0 and v1 have also been removed. Java clients have used message format v2 by default since Kafka 0.11. You can use the kafka.server:type=BrokerTopicMetrics,name={Produce|Fetch}MessageConversionsPerSec metrics to check if any clients are still configured with an older message format (KIP-724) before upgrading.

This release brings several major new features and improvements across all components.

Kafka brokers and clients

Updates to the Kafka broker and clients include the following:

  • The new consumer rebalance protocol introduced in 3.8 is now production ready. It increases the stability and the performance of consumer groups. (KIP-848)
  • Apache Kafka now ships with log4j2. The default configuration file now uses the YAML format. (KIP-653)
  • Support for Queue is in early access. Queues allow multiple consumers to cooperatively read records from the same partition. Records are still consumed by a single consumer in the share group and can be acknowledged individually. (KIP-932)
  • Since Kafka 3.7, Kafka clients can send their metrics to brokers. Now client applications can also inject their own metrics to be sent to brokers. (KIP-1076)
  • Various important improvements around transactions stability (KIP-890), KRaft leader elections with the introduction of pre-votes (KIP-996), and data consistency during leader elections (KIP-966).
  • Clients can proactively re-bootstrap when they hit specific errors or timeouts to avoid being stuck in case all initial bootstrap servers are unreachable. (KIP-1102)

Kafka Connect

Updates to Kafka Connect include the following:

  • The GET /connectors/{connector}/tasks-config endpoint which was deprecated has now been removed. Users should now use the GET /connectors/{connector}/tasks endpoint. (KIP-970)
  • MirrorMaker can now be configured to mirror topics with names ending with .internal or -internal. (KIP-1074)
  • MirrorMaker can now be configured to not mirror heartbeats topics. (KIP-1089)

Kafka Streams

  • The foreign key extractor function in table joins now receives both the record keys and values. (KIP-1104)
  • The new ProcessorWrapper interface enables seamless injection of custom logic around Processor API and DSL processors. (KIP-1112)
  • ProductionExceptionHandler implementations can now retry, gracefully fail or drop problematic records to continue processing efficiently. (KIP-1065)

3.9.1

On March 24, Luke Chen volunteered to be the release manager for Kafka 3.9.1. Since 3.9.0, almost 50 issues have been fixed, including a few blockers. The release is planned for April. You can find the release plan in the wiki.

4.1.0

On March 19, I volunteered to be the release manager for Kafka 4.1.0. The release is planned for July. You can find the release plan in the wiki.

 

Kafka Improvement Proposals

Last month, the community submitted 11 KIPs (KIP-1140 to KIP-1151, KIP-1150 was skipped). I'll highlight a few of them:

  • KIP-1144: Exposing a new public REST API for MirrorMaker 2: When running MirrorMaker in dedicated mode, the Connect REST API is not available. So while this mode was expected to make it easy to run MirrorMaker by hiding Connect and using the arrow notation -> to define mirroring routes, the lack of REST API is a big limitation preventing users from adopting it. This KIP proposes exposing a dedicated REST API specifically tailored to this mode.
  • KIP-1147: Improve consistency of command-line arguments: Kafka has a number of CLI tools that are essential for operators and users to manage their clusters. If you've used the tools you've probably noticed that they don't all use the exact same flags. For example kafka-producer-perf-test.sh uses the --producer-props flag to set producer configurations while kafka-console-producer.sh uses --producer-property. This KIP aims at reviewing all the existing flags and making them consistent across all the CLI tools.
  • KIP-1151: Minimal movement replica balancing algorithm for reassignment: The kafka-reassign-partitions.sh tool allows moving partitions between brokers. It is often used by administrators to balance the load on a cluster. This tool has the --generate flag to generate a partition reassignment. However it's not recommended to use it because the assignment it generates does not take into account the existing assignment. So in most cases, using a generated assignment will lead to a lot of unnecessary replica movements and significant extra load on a cluster. This KIP proposes adding a new flag --rebalance that will aim at minimizing replica movements while still evenly spreading partitions.

Community Releases

  • kroxylicious 0.11: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This new release updates the configuration file format so it's easier to use. It also introduces a new Record Encryption policy to reject producer batches if the proxy can't resolve a key encryption key for a topic.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.