This 72nd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in January 2024.
This is the 6th anniversary of this series! If you are interested in a bit of history about this series, 3 years ago I explained on Twitter how I got started.
For last month’s digest, see Kafka Monthly Digest: December 2023.
The next minor release, 3.7.0, is still in progress:
The release process for Kafka 3.7.0 continued. After a couple of failed release candidates, Stanislav Kozlovski published 3.7.0 RC2 on January 11. A few issues were quickly identified with KRaft (KAFKA-16144, KAFKA-16131), Streams (KAFKA-16141) and JBOD (KAFKA-16157). The community is still working on fixing these. A new release candidate should be available in early February. You can find the release plan in the wiki.
Kafka Improvement Proposals
Last month, the community submitted 5 KIPs (KIP-1014 to KIP-1018). I'll highlight a few of them:
- KIP-1016 Make MM2 heartbeats topic name configurable: The heartbeat connector from MirrorMaker, MirrorHeartbeatConnector, emits heartbeats to help monitor the latency and connectivity between clusters. At the moment, heartbeats are always emitted to a topic called
heartbeatson the source cluster. This KIP aims at making this more flexible by making the topic name configurable.
- KIP-1017: Health check endpoint for Kafka Connect: This KIP proposes introducing a new REST endpoint to Connect,
GET /health, to accurately track the status and health of Connect workers. For example in distributed mode, when workers start they first have to join (or create) the Connect cluster group before being able to run connectors.
- KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests: When consumers fetch records from brokers they provide a maximum amount of time the broker should wait before answering. This is specified by
fetch.max.wait.mswhich defaults to 500 milliseconds. When using Tiered Storage, brokers may require more time to access the remote storage. This KIP proposes introducing a new broker configuration,
fetch.remote.max.wait.ms, to determine the maximum wait time for fetch requests that retrieve records from remote storage.
I selected some interesting blog articles that were published last month:
- You Will Survive Monitoring Kafka Streams
- Seamless data exchange with Kafka Connect and Strimzi on Kubernetes at Decathlon
- Under the hood of Kafka Streams - Tune your storage engine to boost the application performance
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.