Kafka Monthly Digest: December 2024

January 7, 2025
Mickael Maison
Related topics:
Kafka
Related products:
Streams for Apache Kafka

This 83rd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in December 2024. We will also look back at some of the milestones the Apache Kafka project and community reached over the past year, and what's to expect next.

For last month’s digest, see Kafka Monthly Digest: November 2024.

Releases

There is 1 new release and 1 release is in progress:

3.7.2

On December 14, Matthias J. Sax released Kafka 3.7.2. This bugfix release contains 21 fixes including a few blocker issues and several dependency updates. For more details, you can check the release notes and the release plan in the wiki.

4.0.0

Progress on Kafka 4.0.0 continued. Feature freeze was on December 11 and the 4.0 branch was cut December 16. The next milestone is code freeze on January 15. You can find the release plan in the wiki.
 

Kafka Improvement Proposals

Last month, the community submitted 5 KIPs (KIP-1119 to KIP-1123). I'll highlight a couple of them:

  • KIP-1119: Add support for SSL hot reload: Since Kafka 1.1 it's possible to update broker certificates at runtime using either the kafka-configs tool or the Admin API. However it's not possible for clients, and to update their certificates they need to be restarted. This KIP proposes introducing a mechanism enabling Kafka brokers and clients to automatically reload certificates at runtime. This would reduce client-side disruption and also simplify operations for organizations using short-lived certificates.
     
  • KIP-1123: Rack-aware partitioning for Kafka Producer: This KIP proposes updating to default partitioner to support producing to partitions in the same rack as the producer. This can be useful in cases when records don't need to go to specific partitions and when producers are deployed in the same availability zones as brokers as this can lower latency and reduce inter-zone traffic.

Community Releases

  • strimzi-kafka-operator 0.45: Strimzi is a Kubernetes Operator for running Kafka. This new version adds support for Kafka 3.9.0 and 3.8.1. It also provides a mechanism to move data between JBOD disks. It is the last Strimzi minor version that supports ZooKeeper, MirrorMaker1, the Strimzi EnvVar configuration provider, and the Strimzi MirrorMaker2 extensions.
     
  • kroxylicious 0.8.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This release now supports Kafka 3.9. A major internal refactoring is ongoing to improve the state machine for handling connections. Once complete this should allow building filters for new use cases such as authentication offload support. The proxy also now supports mutual TLS authentication with Kafka clients.
     
  • confluent-kafka-javascript 1.0.0: This is a Javascript client based on librdkafka. This new version supports Node v23. It also adds support to fetch offset in the Admin API and improves the KafkaJS Admin API compatibility.

Blogs

I selected some interesting blog articles that were published last month:

Project Milestones in 2024

As the year concludes, let's look at some of the milestones Apache Kafka achieved in 2024:

  • Tiered Storage: After many years of development, Tiered Storage was marked production ready in Kafka 3.9.0. But this is not the end, further improvements are coming in the next releases. For example Kafka 4.0 will enable resizing the remote log manager copier and expiration thread pools at runtime.
  • Build infrastructure: The project moved to GitHub Actions for builds and spent a lot of time stabilizing the test suites and introducing tooling to track and fix flaky tests. While it used to be rare to get a fully green build (all tests passing in all environments), it is now expected for every build.
  • Move to log4j2: The project adopted KIP-653: Upgrade log4j to log4j2 back in 2020 but due to compatibility issues the community decided to delay its implementation to the next major release. When planning Kafka 4.0, work on this KIP restarted and it is now complete. The project also switched to use YAML for the default logging configuration format in 4.0. Older log4j1 configuration files will still be supported but are now deprecated.

Releases in 2024

The project followed its time-based release plan. Consequently, it released three minor versions: 3.7.0, 3.8.0 and 3.9.0 as well as four bugfix releases (3.6.2, 3.7.1, 3.7.2 and 3.8.1). Figure 1 shows the timeline for these releases.

Releases in 2024.
Figure 1. Releases in 2024.

KIPs in 2024

In the past 12 months, the community raised 107 KIPs. Figure 2 shows how this compares to previous years.

KIPs created over the past years.
Figure 2. KIPs created over the past years.

Code and contributors in 2024

This year 61 unique contributors made over 2450 commits. Figure 3 shows the size of the codebase, in lines of code, for a few releases.

Lines of code per Kafka release.
Figure 3. Lines of code per Kafka release.

Committers and PMC in 2024

In 2024, six contributors were invited to become Committers:

  • Igor Soarez
  • Christo Lolov
  • Lianet Magrans
  • Kamal Chandraprakash
  • Jeff Kim
  • Andrew Schofield

Likewise, one Committer also joined the Apache Kafka PMC:

  • Josep Prat

The current roster of Committers and PMC members is available on the Apache Kafka website.

What's coming for Kafka in 2025

First let's look back at the predictions I made last year:

  • Kafka 4.0: Unfortunately Kafka 4.0 is not out yet. To address the last missing features in KRaft mode, the project decided to have a 3.9 release. So 4.0 will release in 2025. As described above, it's well advanced in its release process and should release in the first quarter.
  • KIP-848: The Next Generation of the Consumer Rebalance Protocol: This was first available as preview in Kafka 3.8.0. The current plan is to mark it production ready in Kafka 4.0. Work to adopt the new rebalance protocol in Kafka Streams also started as KIP-1071: Streams Rebalance Protocol was adopted and implementation is ongoing.
  • Removal of Scala code: The amount of Scala in the code base has significantly decreased but it is not down to single digits yet. It is now below 15% while it was around 20% last year. Most of the administration tools (only DumpLogSegments, ConfigCommand and StorageTool are still written in Scala) and a significant part of the storage layer have been migrated. The removal of ZooKeeper is also still ongoing so I expect the number to drop further in the next few weeks.

I'll play the game again and here are my predictions for 2025 (this is my personal point of view and does not represent Apache Kafka nor my employer):

  • KIP-932: Queues for Kafka: The plan is to release this feature as preview in Kafka 4.0. Like many significant features this may need a release or two to fully stabilize. It will be interesting to see how users adopt this feature.
  • KIP-939: Support Participation in 2PC: The KIP has been accepted and work on the implementation has started. There is still a lot of work to do but it is likely this feature will be available, at least in preview, in a release this year. One very interesting use case is the integration with Apache Flink (FLIP-319).
  • Removal of Scala code: I expect the community will complete the tools and log layer rewrite. I also expect work to continue with other components such as networking and request handling logic. With the removal of ZooKeeper, the percentage of Scala in the code base might drop below 10% this year.

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.