Kafka Monthly Digest: September 2021

Welcome to the 44th edition of the Kafka Monthly Digest. In this edition, I'll cover what happened in the Apache Kafka community in September 2021.

For last month’s digest, see Kafka Monthly Digest: August 2021 on IBM Developer.

Releases

Two new Kafka versions (3.0.0 and 2.8.1) were released last month, and work on the next minor release (3.1.0) has also started.

Apache Kafka 3.0

Konstantine Karantasis released version 3.0 on September 21 and published an announcement on the Apache blog. As always, you can find the full list of changes in the release notes or the release plan on the wiki.

Kafka 3.0 is a major milestone in the removal of ZooKeeper (KIP-500). However, like in 2.8, running Kafka without ZooKeeper is still not ready for production.

This new major version deprecates Java 8 and Scala 2.12. Support for these will be removed in the next major release, 4.0.

Kafka 3.0 brings a number of interesting features, which I'll highlight in the next sections.

Brokers and clients

Updates to the Kafka broker and clients include the following:

• There has been lots of progress on the removal of ZooKeeper (KIP-630, KIP-730, KIP-746).
• Old message formats (version 0 and 1) are now deprecated. These versions are used by clients prior to Kafka 0.11 (KIP-724).
• The producer now uses acks=all and enable.idempotence=true by default (KIP-679).
• KafkaFuture can now be converted to CompletionStage for better compatibility with third-party libraries (KIP-707).

Connect

Updates to Kafka Connect include the following:

• MirrorMaker version 1 is now deprecated; users should migrate to the new MirrorMaker (KIP-720).
• Connector client overrides are now enabled by default (KIP-722).
• New API to restart connectors and tasks (KIP-745).

Streams

Updates to Kafka Streams include the following:

• Improved timestamp synchronization to ensure records are processed in timestamp increasing order even when consuming multiple partitions (KIP-695).
• API improvements for tracking task metadata and states (KIP-740, KIP-744).
• Several configuration cleanups such as replication.factor now set to -1 (KIP-733), the default SerDes set to null (KIP-741).
• Several deprecations such as exactly_once and exactly_once_beta in favor of exactly_once_v2 (KIP-732) and the default 24-hour grace period for windows (KIP-633).

Kafka 2.8.1

David Jacot released Kafka 2.8.1 on September 20. This bugfix version addresses 49 Jira issues including a number of blockers. For more details, see the release notes or the release plan.

Kafka 3.1.0

On September 16, David Jacot volunteered to run this minor release. The current target release date is in December. The release plan is available on the wiki.

Kafka Improvement Proposals

Last month, the community submitted eight Kafka Improvement Proposals (KIPs) (KIP-771 to KIP-778). These are the ones that caught my eye:

• KIP-772: Encrypt KRaft Metadata Secrets at Rest: Certain broker configurations are sensitive and need to be stored securely. When overriding broker configurations at runtime, these values are stored in ZooKeeper encrypted. This KIP's goal is to provide a similar mechanism when running in KRaft mode.

• KIP-773: Differentiate consistently metric latency measured in millis and nanos: Kafka metrics tracking durations use the -ns or -ms infix/suffix to describe their unit. This KIP identified a few metrics (bufferpool-wait-time-total, io-waittime-total and iotime-total) that don't follow this naming convention, so it proposes to rename them.

• KIP-777: Improved testability for Admin client: This KIP aims at making the Admin client more testable. Currently, this makes testing logic using Admin pretty verbose. This KIP proposes exposing constructors or factories for Result classes and providing a method to create a failed KafkaFuture.

Community releases

In this section, I will cover releases of notable community projects. This includes only projects that are open source.

• strimzi-kafka-operator 0.25.0: Strimzi is a Kubernetes Operator for running Kafka. This version uses Scala 2.13, contains improvements for SCRAM-SHA-512 users, and adds support for EnvVar Configuration Provider to simplify configuration.

• Librdkafka 1.8.0. Librdkafka is a Kafka client in C/C++. This is a security release as it updates zlib, which had a few CVEs, to 1.2.11. It also contains several fixes around error handling and offset management.

Blogs

Check out these interesting blog articles that were published last month:

• A True Atomic Microservices Implementation with Debezium to Ensure Data Consistency
• Kafka mesh filter in Envoy

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.