Skip to main content
Redhat Developers  Logo
  • Products

    Platforms

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat AI
      Red Hat AI
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • View All Red Hat Products

    Featured

    • Red Hat build of OpenJDK
    • Red Hat Developer Hub
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat OpenShift Local
    • Red Hat Developer Sandbox

      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Secure Development & Architectures

      • Security
      • Secure coding
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • Product Documentation
    • API Catalog
    • Legacy Documentation
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Kafka Monthly Digest: October 2023

November 2, 2023
Mickael Maison

Share:

    This 69th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in October 2023.

    For last month’s digest, see Kafka Monthly Digest: September 2023.

    Releases

    There is a new release: 3.6.0. The release process for 3.7.0 also started.

    3.6.0

    Satish Duggana released Kafka 3.6.0 on October 10 and published an announcement on the Apache Kafka blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.

    This new minor release brings a lot of bug fixes and a few new interesting features, including the long awaited Tiered Storage, which I'll highlight in the next sections.

    Kafka brokers and clients

    Updates to the Kafka broker and clients include the following:

    • Tiered Storage is available in early access. This feature allows moving older log segments to remote storage systems like HDFS or S3. (KIP-405)
    • Brokers can have IPv4 and IPv6 listeners on the same ports. (KIP-797)
    • New configurations, log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms, can be set on topics to validate records' timestamps. (KIP-937)

    Kafka Connect

    Updates to Kafka Connect include the following:

    • New REST API endpoints for updating (PATCH /connectors/{connector}/offsets) and deleting (DELETE /connectors/{connector}/offsets) offsets for connectors. (KIP-875)
    • Connect workers can use the Service Loader mechanism to load connector plugins more efficiently. (KIP-898)

    Kafka Streams

    Updates to Kafka Streams include the following:

    • Stream-Table joins now support a grace period so late records can be processed in timestamp order. (KIP-923)

    3.7.0

    On October 9, Stanislav Kozlovski volunteered to be the release manager for Kafka 3.7.0. You can find the release plan in the wiki.

    Kafka Improvement Proposals

    Last month, the community submitted 13 KIPs (KIP-984 to KIP-996). I'll highlight a few of them:

    • KIP-986: Cross-Cluster Replication: Mirroring clusters is a relatively common use case with Kafka. Today this can be achieved using tools like MirrorMaker. This KIP proposes having a built-in mechanism in brokers to do cross cluster replication.
    • KIP-987: Connect Static Assignments: Kafka Connect makes the assumption that all tasks from connectors are homogeneous. As such it has no options to isolate or treat particularly resource intensive tasks differently. The goal of this KIP is to provide a mechanism for administrators to assign tasks to specific workers to better manage the resources in Connect clusters.
    • KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2 and KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery: Both KIPs aim at improving Kafka Streams Interactive Queries. KIP-985 proposes allowing RangeQueries to return keys in reverse order. KIP-992 aims at solving type safety issues encountered when querying plain and timestamped key-value stores.

    Community Releases

    • Debezium 2.4: Debezium is a Change Data Capture platform. This release introduces ad-hoc blocking snapshots that stop the stream while the snapshot runs. It contains a lot of changes with significant improvements to all connectors. There are also a few breaking changes that are worth checking before you upgrade.
    • strimzi-kafka-operator 0.38: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.6. It also includes improvements for scaling down operations, and Connect and MirrorMaker connectors can now be stopped.
    • Librdkafka 2.3: Librdkafka is a Kafka client in C/C++. It adds support for describing clusters and topics via the Admin API. It also contains many fixes and enhancements in the idempotent producer and consumer clients.

    Blogs

    I selected some interesting blog articles that were published last month:

    • The advantages of queues on logs
    • Developing Kafka client applications: A simple producer
    • A Size for Every Stream: The Expert's Guide to Sizing Kafka Streams
    • How to 'Kstreamplify' : your new way to develop Kafka Streams application

    To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

     

    Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.

    Recent Posts

    • The secure way to handle secrets in OpenShift

    • How to deploy MCP servers on OpenShift using ToolHive

    • How to change the meaning of python and python3 on RHEL

    • vLLM or llama.cpp: Choosing the right LLM inference engine for your use case

    • How to implement and monitor circuit breakers in OpenShift Service Mesh 3

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Platforms

    • Red Hat AI
    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue