This 71st edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in December 2023. We will also look back at some of the milestones that the Kafka project and community reached over the past year.
For last month’s digest, see Kafka Monthly Digest: November 2023.
There are 2 new releases 3.6.1 and 3.5.2, and 3.7.0 is still in progress:
The release process for Kafka 3.7.0 continued. Feature freeze happened on December 6 and the 3.7 branch was cut on December 12. The first release candidate should be ready in the next few days. The release date is still targeted for January. You can find the release plan in the wiki.
On December 7, I released Kafka 3.6.1. This bugfix release updates the ZooKeeper and Netty dependencies to address CVEs and also contain ~20 fixes. For more details, check the announcement on the blog and the release plan on the wiki.
On December 11, Luke Chen released Kafka 3.5.2. This bugfix release updates a few dependencies to address CVEs and also includes 39 fixes. You can find the announcement on the blog and the release plan on the wiki.
Kafka Improvement Proposals
Last month, the community submitted 4 KIPs (KIP-1010 to KIP-1013). I'll highlight a few of them:
- KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh: Since Kafka 2.3, the
IncrementalAlterConfigsAPI is preferred over the now deprecated
AlterConfigsAPI to update broker, user or quota configurations at runtime. This KIP aims at updating the
kafka-configscommand line tool to use the
IncrementalAlterConfigsAPI for all operations as today
AlterConfigsis still used when updating broker configurations.
KIP-1012: The need for a Kafka 3.8.x release: KIP-833 had established a roadmap to Kafka 4.0 but several things have changed since 2022. The goal of this new KIP is to determine what needs to be done before releasing Kafka 4.0 which will completely remove ZooKeeper.
KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7): Kafka only updates the Java versions it supports with major releases. For example Kafka 2.0 dropped support for Java 7. This KIP proposes dropping support for Java 11 for the broker and tools in Kafka 4.0. Note that Kafka 4.0 will drop support for Java 8 due to KIP-750.
- kroxylicious 0.4.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This first release includes a preview of an
EnvelopeEncryptionfilter which provides transparent encryption at rest.
- Debezium 2.5: Debezium is a Change Data Capture platform. The new release includes improvements in all connectors. It introduces a new connector for IBM Informix, and brings preview support for MariaDB with the goal to bring it in line with the MySQL connector. There are also a few breaking changes that are worth checking before you upgrade.
- strimzi-kafka-operator 0.39: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.5.2 and 3.6.1. The
UnidirectionalTopicOperatorfeature gates are now GA and enabled by default. The example Grafana dashboards for Kafka Connect and MirrorMaker have also been updated. Finally the Strimzi project introduced a new component, strimzi-mqtt-bridge, which provides an MQTT bridge to Apache Kafka.
I selected some interesting blog articles that were published last month:
- Processing XML with Kafka Connect
- Kafka Streams Sightings in the Wild
- KEDA + Kafka: Improve performance by 62.15% at peak loads
- Kafka: Automating Root CA rotation with Vault
- Streamlined Performance: Debezium JDBC connector batch support
Project Milestones in 2023
As the year concludes, let's look at some of the milestones Apache Kafka achieved in 2023.
Releases in 2023
The project followed its time-based release plan. Consequently, it released three minor versions: 3.4, 3.5 and 3.6 as well as five bugfix releases (3.3.2, 3.4.1, 3.5.1, 3.5.2 and 3.6.1). Figure 1 shows the timeline for these releases.
In the past 12 months the community raised 116 KIPs, crossing the 1000th KIP mark in November. Figure 2 shows how this compares to previous years.
Code and contributors
This year 59 unique contributors made more than 1100 commits. Figure 3 shows the size of the code base, in lines of code, for a few releases.
Committers and Project Management Committee members
In 2023, ten contributors were invited to become Committers:
- Satish Duggana
- Justine Olshan
- Edoardo Comar
- Walker Carlson
- Stanislav Kozlovski
- Lucas Bradstreet
- Divij Vaidya
- Greg Harris
- Lucas Brutschy
- Yash Mayya
Likewise, five Committers also joined the Apache Kafka PMC:
- Chris Egerton
- David Arthur
- Justine Olshan
- Satish Duggana
- Divij Vaidya
In June, Jun Rao stepped down from his role as chair of the Apache Kafka PMC. I was elected by the PMC to become the new chair.
The current roster of Committers and PMC members is available on the Kafka website.
What's coming for Kafka in 2024
Last year I predicted that Kafka in 2023 would get support for Tiered Storage, have a migration process from ZooKeeper to KRaft and have a new consumer rebalance protocol. Tiered Storage finally landed in early access in 3.6.0, migrating from ZooKeeper to KRaft initially landed in 3.4.0 and was marked as production ready in 3.6.0. A lot of work has happened on the new consumer rebalance protocol but it's still not available.
I'll play the game again and here are my predictions for 2024 (this is my personal point of view and does not represent Apache Kafka nor my employer)
- Kafka 4.0: KIP-500 which proposed replacing ZooKeeper was voted over 4 years ago. Changing the underlying metadata coordination layer is definitively not a simple task! At last it seems all the requirements to fully remove ZooKeeper are almost there and I expect Kafka 4.0 to release this year. It will be interesting to see how many lines of codes this will remove from the code base.
- KIP-848: The Next Generation of the Consumer Rebalance Protocol: Looking at KAFKA-14048 over two thirds of the tasks are now completed so it's likely the new consumer rebalance protocol will be introduced in 2024.
- Removal of Scala code: Kafka was initially mostly written in Scala. Over the years every time a new module was created, the community used Java. Scala now accounts for less than 20% of the code base. I expect Kafka to still have some Scala in the next few years but with the removal of ZooKeeper and work on-going to split the last Scala module, the percentage of Scala in the code base may drop into single digits in 2024.
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.