Kafka Monthly Digest: October 2023

This 69th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in October 2023.

For last month’s digest, see Kafka Monthly Digest: September 2023.

Releases

There is a new release: 3.6.0. The release process for 3.7.0 also started.

3.6.0

Satish Duggana released Kafka 3.6.0 on October 10 and published an announcement on the Apache Kafka blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.

This new minor release brings a lot of bug fixes and a few new interesting features, including the long awaited Tiered Storage, which I'll highlight in the next sections.

Kafka brokers and clients

Updates to the Kafka broker and clients include the following:

• Tiered Storage is available in early access. This feature allows moving older log segments to remote storage systems like HDFS or S3. (KIP-405)
• Brokers can have IPv4 and IPv6 listeners on the same ports. (KIP-797)
• New configurations, log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms, can be set on topics to validate records' timestamps. (KIP-937)

Kafka Connect

Updates to Kafka Connect include the following:

• New REST API endpoints for updating (PATCH /connectors/{connector}/offsets) and deleting (DELETE /connectors/{connector}/offsets) offsets for connectors. (KIP-875)
• Connect workers can use the Service Loader mechanism to load connector plugins more efficiently. (KIP-898)

Kafka Streams

Updates to Kafka Streams include the following:

• Stream-Table joins now support a grace period so late records can be processed in timestamp order. (KIP-923)

3.7.0

On October 9, Stanislav Kozlovski volunteered to be the release manager for Kafka 3.7.0. You can find the release plan in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 13 KIPs (KIP-984 to KIP-996). I'll highlight a few of them:

• KIP-986: Cross-Cluster Replication: Mirroring clusters is a relatively common use case with Kafka. Today this can be achieved using tools like MirrorMaker. This KIP proposes having a built-in mechanism in brokers to do cross cluster replication.
• KIP-987: Connect Static Assignments: Kafka Connect makes the assumption that all tasks from connectors are homogeneous. As such it has no options to isolate or treat particularly resource intensive tasks differently. The goal of this KIP is to provide a mechanism for administrators to assign tasks to specific workers to better manage the resources in Connect clusters.
• KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2 and KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery: Both KIPs aim at improving Kafka Streams Interactive Queries. KIP-985 proposes allowing RangeQueries to return keys in reverse order. KIP-992 aims at solving type safety issues encountered when querying plain and timestamped key-value stores.

Community Releases

• Debezium 2.4: Debezium is a Change Data Capture platform. This release introduces ad-hoc blocking snapshots that stop the stream while the snapshot runs. It contains a lot of changes with significant improvements to all connectors. There are also a few breaking changes that are worth checking before you upgrade.
• strimzi-kafka-operator 0.38: Strimzi is a Kubernetes Operator for running Kafka. This release adds support for Kafka 3.6. It also includes improvements for scaling down operations, and Connect and MirrorMaker connectors can now be stopped.
• Librdkafka 2.3: Librdkafka is a Kafka client in C/C++. It adds support for describing clusters and topics via the Admin API. It also contains many fixes and enhancements in the idempotent producer and consumer clients.

Blogs

I selected some interesting blog articles that were published last month:

• The advantages of queues on logs
• Developing Kafka client applications: A simple producer
• A Size for Every Stream: The Expert's Guide to Sizing Kafka Streams
• How to 'Kstreamplify' : your new way to develop Kafka Streams application

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.