Kafka Monthly Digest: January 2024

January 31, 2024
Mickael Maison

This 72nd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in January 2024.

This is the 6th anniversary of this series! If you are interested in a bit of history about this series, 3 years ago I explained on Twitter how I got started.

For last month’s digest, see Kafka Monthly Digest: December 2023.

Releases

The next minor release, 3.7.0, is still in progress:

3.7.0

The release process for Kafka 3.7.0 continued. After a couple of failed release candidates, Stanislav Kozlovski published 3.7.0 RC2 on January 11. A few issues were quickly identified with KRaft (KAFKA-16144, KAFKA-16131), Streams (KAFKA-16141) and JBOD (KAFKA-16157). The community is still working on fixing these. A new release candidate should be available in early February. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 5 KIPs (KIP-1014 to KIP-1018). I'll highlight a few of them:

  • KIP-1016 Make MM2 heartbeats topic name configurable: The heartbeat connector from MirrorMaker, MirrorHeartbeatConnector, emits heartbeats to help monitor the latency and connectivity between clusters. At the moment, heartbeats are always emitted to a topic called heartbeats on the source cluster. This KIP aims at making this more flexible by making the topic name configurable.
  • KIP-1017: Health check endpoint for Kafka Connect: This KIP proposes introducing a new REST endpoint to Connect, GET /health, to accurately track the status and health of Connect workers. For example in distributed mode, when workers start they first have to join (or create) the Connect cluster group before being able to run connectors.
  • KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests: When consumers fetch records from brokers they provide a maximum amount of time the broker should wait before answering. This is specified by fetch.max.wait.ms which defaults to 500 milliseconds. When using Tiered Storage, brokers may require more time to access the remote storage. This KIP proposes introducing a new broker configuration, fetch.remote.max.wait.ms, to determine the maximum wait time for fetch requests that retrieve records from remote storage.

Community Releases

None of the projects I follow have made a release in January. If you know a popular open source community project that I don't cover, please let me know on Twitter or Mastodon.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Last updated: February 1, 2024
Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.