Skip to main content
AI
  • Get started with AI

    • Red Hat AI

      Accelerate the development and deployment of enterprise AI solutions.

    • AI learning hub

      Explore learning materials and tools, organized by task.

    • AI interactive demos

      Click through scenarios with Red Hat AI, including training LLMs and more.

    • AI/ML learning paths

      Expand your Red Hat OpenShift AI knowledge using these learning resources.

    • AI quickstarts

      Focused AI use cases designed for fast deployment on Red Hat AI platforms.

    • No-cost AI training

      Foundational Red Hat AI training.

  • Featured resources

    • Red Hat OpenShift AI
    • Open source AI for developers
    • AI product application development
    • Open source-powered AI/ML for hybrid cloud
    • AI and Node.js cheat sheet

Red Hat AI Factory with NVIDIA

Red Hat AI Factory with NVIDIA is a co-engineered, enterprise-grade AI solution for building, deploying, and managing AI at scale across hybrid cloud environments.

Learn
  • Self-guided

    • Documentation

      Find answers, get step-by-step guidance, and learn how to use Red Hat products.

    • Learning paths

      Explore curated walkthroughs for common development tasks.

    • Guided learning

      Receive custom learning paths powered by our AI assistant.

    • See all learning
  • Hands-on

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

    • Interactive labs

      Learn by doing in these hands-on, browser-based experiences.

    • Interactive demos

      Click through product features in these guided tours.

  • Browse by topic

    • AI/ML
    • Automation
    • Java
    • Kubernetes
    • Linux
    • See all topics
  • Training & certifications

    • Courses and exams
    • Certifications
    • Skills assessments
    • Red Hat Academy
    • Learning subscription
    • Explore training
Build
  • Get started

    • Red Hat build of Podman Desktop

      A downloadable, local development hub to experiment with our products and builds.

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

  • Download products

    • Red Hat Enterprise Linux
    • Red Hat AI
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products
  • Featured

    • Red Hat build of OpenJDK
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat Developer Toolset
  • References

    • E-books
    • Documentation
    • Cheat sheets
    • Architecture center
Community
  • Get involved

    • Events
    • Live AI events
    • Red Hat Summit
    • Red Hat Accelerators
    • Community discussions
  • Follow along

    • Articles & blogs
    • Developer newsletter
    • Videos
    • Github
  • Get help

    • Customer service
    • Customer support
    • Regional contacts
    • Find a partner

Join the Red Hat Developer program

Download Red Hat products and project builds, access support documentation, learning content, and more.

ConsoleDocsSupport Search
New For you

Recommended

We'll recommend resources you may like as you browse. Try these suggestions for now.

  • Product downloads
  • Cheat sheets
  • Red Hat Developer learning portal
  • Developer blog
  • E-books
  • Developer Sandbox
Log in

Get more with a Red Hat Developer account

  • Product downloads
  • Console access
  • Event registration
  • Training & trials
Explore more account benefitsLog in or register

A subscription may be required for some services.

Contact us

Kafka Monthly Digest: November 2024

December 2, 2024
Mickael Maison
Related topics:
Kafka
Related products:
Streams for Apache Kafka

    This 82nd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in November 2024.

    For last month’s digest, see Kafka Monthly Digest: October 2024.

    Releases

    There is 1 new release and 2 releases are in progress:

    3.9.0

    On November 8, Colin McCabe released Apache Kafka 3.9.0. You can find the announcement on the Apache Kafka blog. You can also check the release notes and the release plan in the wiki for the full details.

    This is the final minor release in the 3.X line. Kafka 4.0 will no longer support ZooKeeper, so clusters still using ZooKeeper must migrate to KRaft using 3.9 before being able to upgrade to 4.0.

    This new minor release brings several new features and many bug fixes.

    Kafka brokers and client

    Updates to the Kafka broker and clients include the following:

    • Tiered storage is now production ready. You can enable and disable it per topic and set network quotas. (KIP-950, KIP-956, KIP-1005, KIP-1057)
    • New dynamic KRaft quorums. This allows adding and removing controller nodes at runtime. Until now KRaft quorums required to define all controllers upfront and they could not change. From 3.9 the recommended deployment method is to use a dynamic quorum. However note that Kafka currently does not provide a way to update existing static quorums to be dynamic, so only new quorums deployed with Kafka 3.9 can be dynamic. (KIP-853)
    • New configuration, sasl.oauthbearer.header.urlencode, to make the OAuth authentication with an OIDC provider compliant with RFC-6749. (KIP-1024) 

    Kafka Connect

    Updates to Kafka Connect include the following:

    • New REST endpoint, GET /health to check the state of a Connect worker. (KIP-1017)
    • Consumer offset translation can now be fully disabled by setting emit.offset-syncs.enabled=false in MirrorSourceConnector's configuration. (KIP-1031)
    • The InsertField, ExtractField, HeaderFrom, Cast, SetSchemaMetadata, TimestampConverter and MaskField transformations now support the replace.null.with.default configuration (like JsonConverter) to better handle null values. (KIP-1040)

    Kafka Streams

    • New ProcessingExceptionHandler interface to simplify the handling of processing errors. (KIP-1033)
    • A new configuration, log.summary.interval.ms, allows setting how often, or even disable if set to 0, the summary information is logged. (KIP-1049)
       

    4.0.0

    Progress on Kafka 4.0.0 continued. KIP freeze happened on November 20. The next milestone is feature freeze on December 11. You can find the release plan in the wiki.

    3.7.2

    All the fixes we wanted to backport for Kafka 3.7.2 have been merged. We should expect it to release in December. You can find the release plan in the wiki.

    Kafka Improvement Proposals

    Last month, the community submitted 13 KIPs (KIP-1106 to KIP-1118). I'll highlight a few of them:

    • KIP-1109: Unifying Kafka Consumer Topic Metrics: For historical reasons, dots in topic names are replaced by underscores in the Kafka Consumer metrics. This can lead to confusion and it is not consistent with other metrics, for example from the Producer, which preserve topic names. This KIP aims at fixing this inconsistency by keeping topic names unchanged in the Consumer metrics.
       
    • KIP-1111: Enforcing Explicit Naming for Kafka Streams Internal Topics: Kafka Streams creates internal topics to store state. The names of these topics are computed when the Streams application starts, and while they follow a pattern it's not always easy to guess what the actual names will be for complex topologies. This KIP proposes a mechanism so users can explicitly provide the names of all internal topics used by a Kafka Streams application.
       
    • KIP-1117: Support keystore with multiple alias entries: When a keystore is provided as part of the TLS configuration, Kafka always picks the first key. This KIP proposes introducing a new configuration, ssl.keystore.alias, to explicitly select the key to use in a keystore.

    Community Releases

    • node-rdkafka 3.2. Node-rdkafka is a Node.js Kafka client based on librdkafka. This new release is now based on librdkafka 2.6.0 and adds support for cooperative assignment in the Consumer.

    Blogs

    I selected some interesting blog articles that were published last month:

    • Auto-rebalancing on cluster scaling
    • Dynamic Kafka controller quorum
    • Deep Dive into Handling Consumer Fetch Requests: Kafka Producer and Consumer Internals, Part 4

    To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

    Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.

    Recent Posts

    • What's new for developers in Red Hat Enterprise Linux 10.3 and 9.9

    • Control multicluster costs with right-sizing in Red Hat Advanced Cluster Management 5

    • Lightbits is now certified for Red Hat OpenStack Services on OpenShift

    • What's new in Red Hat OpenShift 5 for developers

    • On-demand screen context in Red Hat Developer Hub intelligent assistant

    Red Hat Developers logo

    Platforms

    • Red Hat AI
    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products

    Build

    • Developer Sandbox
    • Developer tools
    • Interactive tutorials
    • API catalog

    Quicklinks

    • Learning resources
    • E-books
    • Cheat sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site status dashboard
    • Report a security problem

    Red Hat Developer

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2026 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility
    Ask AI