Kafka Monthly Digest: February 2024

March 1, 2024
Mickael Maison

This 73rd edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in February 2024.

For last month’s digest, see Kafka Monthly Digest: January 2024.

Releases

There is a new Kafka release: 3.7.0:

3.7.0

To address the issues identified with Kafka 3.7.0 RC2, Stanislav Kozlovski published 3.7.0 RC4 on February 9. The vote passed and 3.7.0 released on February 27. You can find the announcement on the Apache Kafka blog. You can also check the release notes and the release plan in the wiki.

This new minor release brings a lot of bug fixes and several new interesting features, including support for JBOD in KRaft in early access and support for the new consumer group protocol also in early access.

Kafka brokers and clients

Updates to the Kafka broker and clients include the following:

  • JBOD (Just a Bunch of Disk) support in KRaft is available in early access. JBOD allows using several disks to store Kafka logs. This is not ready for production usage yet, you can check the JBOD specific release notes for details about the current state of this feature. (KIP-858)
  • The new consumer rebalance protocol is also available in early access. This new protocol moves most of the rebalance logic into the coordinator instead of clients. It keeps the same semantics for consumer groups but it is more efficient and should perform better. Check the specific release notes for this features for details about its current state. (KIP-848)
  • Clients can now send their metrics to brokers. This allows to easily collect client metrics and simplifies the monitoring the client-side applications. (KIP-714)
  • Apache Kafka now provides an official Docker image on Dockerhub. (KIP-975)

Kafka Connect

Updates to Kafka Connect include the following:

  • New built-in BooleanConverter. (KIP-959)
  • The /admin/loggers endpoint can now adjust log levels across all workers in a Connect cluster. (KIP-976)
  • Connectors can now be created in the stopped state. This is useful to allow setting initial offsets before starting a connector. (KIP-980)

Kafka Streams

Updates to Kafka Streams include the following:

  • Support for rack-aware task assignments. (KIP-925)
  • Null-key records are now allowed in some joins. (KIP-962)
  • A lot of improvements to the interactive queries v2 (IQv2) API, including support for versioned state stores (KIP-960, KIP-968), the ability to retrieve results in ascending or descending order (KIP-985)

3.8.0

On February 26, Josep Prat volunteered to be the release manager for Kafka 3.8.0. The release plan is available in the wiki.

Kafka Improvement Proposals

Last month, the community submitted 6 KIPs (KIP-1019 to KIP-1024). I'll highlight a few of them:

  • KIP-1019: Expose method to determine Metric Measurability: The KafkaMetric object is used to define metrics in the Kafka brokers and clients. It can contain a Gauge to retrieve an instant value, or a Measurable that tracks a quantity. Today there isn't a method to  determine what a KafkaMetric contains, you need to try calling its measurable() method and either get a Measurable or catch an exception if it contained a Gauge. This KIP proposes adding a new method, isMeasurable(), to solve this issue.
  • KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh: The kafka-get-offsets.sh tool allows retrieving the offset from a topic partition for a specified timestamp with the --time flag. It also supports retrieving important offsets such as the latest, earliest, max timestamp offsets using special negative values. When retrieving the latest offset, the tool currently returns the "high watermark" which is the latest offset that has been replicated to all the in-sync replicas. When using transactions, it can also be useful to find the last stable offset (LSO) which is is the latest offset that has been replicated to all the in-sync replicas and that is not in not part of an open transaction. Consumers configured with isolation.level=read_committed can consume records only up to the LSO. This KIP updates the kafka-get-offsets.sh to return the LSO if the --time flag is set to -6.
  • KIP-1022: Formatting and Updating Features: As Kafka gained features over time, at some point it became important to clearly define which features and their specific versions each broker supports. This is necessary for clients to ensure they make the best usage of the available APIs but also between brokers as some features can only be enabled once all brokers in the cluster support it. KIP-584 introduced a few years ago a versioning scheme for features that was tied to the broker metadata version. This KIP aims at making this more flexible by allowing to update the version of specific features via the kafka-storage.sh and kafka-features.sh tools.

Community Releases

  • Sarama 1.43: Sarama is a pure Golang Kafka client. This version bumps the minimum supported Go version to 1.19 and adds support for providing a custom Service Principal Name when using Kerberos authentication.

Blogs

I selected some interesting blog articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.

Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.