Kafka Monthly Digest: October 2023

November 2, 2023
Mickael Maison

This 69th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in October 2023.

For last month’s digest, see Kafka Monthly Digest: September 2023.

Releases

There is a new release: 3.6.0. The release process for 3.7.0 also started.

3.6.0

Satish Duggana released Kafka 3.6.0 on October 10 and published an announcement on the Apache Kafka blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.

This new minor release brings a lot of bug fixes and a few new interesting features, including the long awaited Tiered Storage, which I'll highlight in the next sections.

Kafka brokers and clients

Updates to the Kafka broker and clients include the following:

  • Tiered Storage is available in early access. This feature allows moving older log segments to remote storage systems like HDFS or S3. (KIP-405)
  • Brokers can have IPv4 and IPv6 listeners on the same ports. (KIP-797)
  • New configurations, log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms, can be set on topics to validate records' timestamps. (KIP-937)

Kafka Connect

Updates to Kafka Connect include the following:

  • New REST API endpoints for updating (PATCH /connectors/{connector}/offsets) and deleting (DELETE /connectors/{connector}/offsets) offsets for connectors. (KIP-875)
  • Connect workers can use the Service Loader mechanism to load connector plugins more efficiently. (KIP-898)

Kafka Streams

Updates to Kafka Streams include the following:

  • Stream-Table joins now support a grace period so late records can be processed in timestamp order. (KIP-923)

3.7.0

On October 9, Stanislav Kozlovski volunteered to be the release manager for Kafka 3.7.0. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 13 KIPs (KIP-984 to KIP-996). I'll highlight a few of them:

Community Releases

  • Debezium 2.4: Debezium is a Change Data Capture platform. This release introduces ad-hoc blocking snapshots that stop the stream while the snapshot runs. It contains a lot of changes with significant improvements to all connectors. There are also a few breaking changes that are worth checking before you upgrade.
  • strimzi-kafka-operator 0.38: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.6. It also includes improvements for scaling down operations, and Connect and MirrorMaker connectors can now be stopped.
  • Librdkafka 2.3: Librdkafka is a Kafka client in C/C++. It adds support for describing clusters and topics via the Admin API. It also contains many fixes and enhancements in the idempotent producer and consumer clients.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

 

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.