Skip to main content
Redhat Developers  Logo
  • Products

    Platforms

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat AI
      Red Hat AI
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • View All Red Hat Products

    Featured

    • Red Hat build of OpenJDK
    • Red Hat Developer Hub
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat OpenShift Local
    • Red Hat Developer Sandbox

      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Secure Development & Architectures

      • Security
      • Secure coding
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • Product Documentation
    • API Catalog
    • Legacy Documentation
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Kafka Monthly Digest: December 2023

December 22, 2023
Mickael Maison

Share:

    This 71st edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in December 2023. We will also look back at some of the milestones that the Kafka project and community reached over the past year.

    For last month’s digest, see Kafka Monthly Digest: November 2023.

    Releases

    There are 2 new releases 3.6.1 and 3.5.2, and 3.7.0 is still in progress:

    3.7.0

    The release process for Kafka 3.7.0 continued. Feature freeze happened on December 6 and the 3.7 branch was cut on December 12. The first release candidate should be ready in the next few days. The release date is still targeted for January. You can find the release plan in the wiki.

    3.6.1

    On December 7, I released Kafka 3.6.1. This bugfix release updates the ZooKeeper and Netty dependencies to address CVEs and also contain ~20 fixes. For more details, check the announcement on the blog and the release plan on the wiki.

    3.5.2

    On December 11, Luke Chen released Kafka 3.5.2. This bugfix release updates a few dependencies to address CVEs and also includes 39 fixes. You can find the announcement on the blog and the release plan on the wiki.

    Kafka Improvement Proposals

    Last month, the community submitted 4 KIPs (KIP-1010 to KIP-1013). I'll highlight a few of them:

    • KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh: Since Kafka 2.3, the IncrementalAlterConfigs API is preferred over the now deprecated AlterConfigs API to update broker, user or quota configurations at runtime. This KIP aims at updating the kafka-configs command line tool to use the IncrementalAlterConfigs API for all operations as today AlterConfigs is still used when updating broker configurations.
    • KIP-1012: The need for a Kafka 3.8.x release: KIP-833 had established a roadmap to Kafka 4.0 but several things have changed since 2022. The goal of this new KIP is to determine what needs to be done before releasing Kafka 4.0 which will completely remove ZooKeeper.

    • KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7): Kafka only updates the Java versions it supports with major releases. For example Kafka 2.0 dropped support for Java 7. This KIP proposes dropping support for Java 11 for the broker and tools in Kafka 4.0. Note that Kafka 4.0 will drop support for Java 8 due to KIP-750.

    Community Releases

    • kroxylicious 0.4.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This first release includes a preview of an EnvelopeEncryption filter which provides transparent encryption at rest.
    • Debezium 2.5: Debezium is a Change Data Capture platform. The new release includes improvements in all connectors. It introduces a new connector for IBM Informix, and brings preview support for MariaDB with the goal to bring it in line with the MySQL connector. There are also a few breaking changes that are worth checking before you upgrade.
    • strimzi-kafka-operator 0.39: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.5.2 and 3.6.1. The StableConnectIdentities, KafkaNodePools and UnidirectionalTopicOperator feature gates are now GA and enabled by default. The example Grafana dashboards for Kafka Connect and MirrorMaker have also been updated. Finally the Strimzi project introduced a new component, strimzi-mqtt-bridge, which provides an MQTT bridge to Apache Kafka.

    Blogs

    I selected some interesting blog articles that were published last month:

    • Processing XML with Kafka Connect
    • Kafka Streams Sightings in the Wild
    • KEDA + Kafka: Improve performance by 62.15% at peak loads
    • Kafka: Automating Root CA rotation with Vault
    • Streamlined Performance: Debezium JDBC connector batch support

    Project Milestones in 2023

    As the year concludes, let's look at some of the milestones Apache Kafka achieved in 2023.

    Releases in 2023

    The project followed its time-based release plan. Consequently, it released three minor versions: 3.4, 3.5 and 3.6 as well as five bugfix releases (3.3.2, 3.4.1, 3.5.1, 3.5.2 and 3.6.1). Figure 1 shows the timeline for these releases.

    Figure 1. Releases in 2023.
    Figure 1. Releases in 2023.

    KIPs

    In the past 12 months the community raised 116 KIPs, crossing the 1000th KIP mark in November. Figure 2 shows how this compares to previous years.

    KIPs created over the past years
    Figure 2. KIPs created over the past years.
    Figure 2. KIPs created over the past years.

    Code and contributors

    This year 59 unique contributors made more than 1100 commits. Figure 3 shows the size of the code base, in lines of code, for a few releases.

    Lines of codes across Apache Kafka releases
    Figure 3. Lines of code per Kafka release.

    Committers and Project Management Committee members

    In 2023, ten contributors were invited to become Committers:

    • Satish Duggana
    • Justine Olshan
    • Edoardo Comar
    • Walker Carlson
    • Stanislav Kozlovski
    • Lucas Bradstreet
    • Divij Vaidya
    • Greg Harris
    • Lucas Brutschy
    • Yash Mayya

    Likewise, five Committers also joined the Apache Kafka PMC:

    • Chris Egerton
    • David Arthur
    • Justine Olshan
    • Satish Duggana
    • Divij Vaidya

    In June, Jun Rao stepped down from his role as chair of the Apache Kafka PMC. I was elected by the PMC to become the new chair.

    The current roster of Committers and PMC members is available on the Kafka website.

    What's coming for Kafka in 2024

    Last year I predicted that Kafka in 2023 would get support for Tiered Storage, have a migration process from ZooKeeper to KRaft and have a new consumer rebalance protocol. Tiered Storage finally landed in early access in 3.6.0, migrating from ZooKeeper to KRaft initially landed in 3.4.0 and was marked as production ready in 3.6.0. A lot of work has happened on the new consumer rebalance protocol but it's still not available.

    I'll play the game again and here are my predictions for 2024 (this is my personal point of view and does not represent Apache Kafka nor my employer)

    • Kafka 4.0: KIP-500 which proposed replacing ZooKeeper was voted over 4 years ago. Changing the underlying metadata coordination layer is definitively not a simple task! At last it seems all the requirements to fully remove ZooKeeper are almost there and I expect Kafka 4.0 to release this year. It will be interesting to see how many lines of codes this will remove from the code base.
    • KIP-848: The Next Generation of the Consumer Rebalance Protocol: Looking at KAFKA-14048 over two thirds of the tasks are now completed so it's likely the new consumer rebalance protocol will be introduced in 2024.
    • Removal of Scala code: Kafka was initially mostly written in Scala. Over the years every time a new module was created, the community used Java. Scala now accounts for less than 20% of the code base. I expect Kafka to still have some Scala in the next few years but with the removal of ZooKeeper and work on-going to split the last Scala module, the percentage of Scala in the code base may drop into single digits in 2024.

    To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

    Last updated: January 2, 2024
    Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.

    Recent Posts

    • How to change the meaning of python and python3 on RHEL

    • vLLM or llama.cpp: Choosing the right LLM inference engine for your use case

    • How to implement and monitor circuit breakers in OpenShift Service Mesh 3

    • Analysis of OpenShift node-system-admin-client lifespan

    • What's New in OpenShift GitOps 1.18

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Platforms

    • Red Hat AI
    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue