Apache Kafka excels at moving data within a cluster. Leaders replicate to followers, consumers pull from any replica, and the entire machinery runs with minimal operational overhead. Moving data between clusters has never been that simple.

Organizations run multiple Kafka clusters for good reasons: Geographic distribution, compliance boundaries, team isolation, version segregation. But once data lands in a cluster, getting a faithful copy into another has always required external tooling, careful coordination, and a healthy tolerance for operational surprises.

KIP-1279 changes this. Cluster mirroring embeds cross-cluster replication directly into the Kafka broker. No external processes, no offset translation tables, no recompression. A destination broker fetches committed records from a source cluster using the same fetch protocol that followers already use, and appends them to local partition logs byte for byte. The result is a mirror that preserves offsets, compression, and consumer group state, making failover as simple as stopping the mirror and redirecting clients.

This article covers the high-level architecture behind cluster mirroring, the state machine that governs mirror partitions, and the consistency guarantees that hold it all together. I then walk through 2 practical scenarios (disaster recovery and cluster migration), and end with a video demo.

A new approach cross-cluster replication

MirrorMaker 2 (MM2) has served as the standard tool for cross-cluster replication since Kafka 2.4, running as a set of Kafka Connect workers that consume from a source cluster and produce to a destination cluster. Cluster mirroring takes a fundamentally different approach by embedding replication directly into the broker.

• Zero external infrastructure: Mirror fetcher threads run inside the broker process itself. There are no Connect workers to provision, monitor, or scale. A single CLI command (kafka-cluster-mirrors.sh --create) establishes a mirror; another (--start) begins replicating topics. The entire lifecycle is managed through the same Admin API used for topics and consumer groups.
• Byte-for-byte transfer: Compressed batches are replicated as raw bytes, never decompressing or recompressing them. A gzip, snappy, lz4, or zstd batch arrives at the destination in its original form. This eliminates the CPU overhead of a decompress/recompress round trip and preserves the producer's original compression choices.
• Exact offset preservation: The destination log maintains the same offsets as the source, including gaps left by topic compaction. Consumer groups fail over without offset translation: The committed offset on the source is the committed offset on the destination.
• One-command failover: Stopping a mirror (--stop) transitions partitions through a deterministic sequence: fetchers are removed, the last mirror epoch is persisted, the leader epoch is bumped, pending transactions are aborted, and a new control record expires all producer state. The partition becomes writable on the destination. No external coordination or offset queries required.
• Unclean leader election support: When the source cluster experiences an unclean leader election (ULE), the destination enters a recovery state. It waits for all assigned replicas, not just ISR members, to converge to the truncated offset before resuming replication. This ensures log consistency between clusters even when the source elects a leader with an incomplete log.

The following table summarizes the key differences:

With cluster mirroring, destination brokers become active participants in cross-cluster replication. Each one fetches data directly from the source cluster using the standard fetch protocol and appends raw record batches to local partition logs. Source and destination partitions share the same topic ID.

Beyond data replication, the broker also handles metadata discovery, configuration syncing, groups offset syncing, and ACL propagation. Bandwidth control works on both sides. The destination broker enforces a configurable replication rate limit. On the source side, mirror fetch traffic presents as standard consumer requests, so existing client quota mechanisms apply without modification.

Architecture

There are 3 main components collaborating within each destination broker. Figure 1 shows how they connect to each other and to the source cluster.

MirrorMetadataManager

MirrorMetadataManager (MMM) is the orchestrator. Running on every broker, it implements the MetadataPublisher interface to react to changes in the KRaft metadata log. When the controller writes a MirrorTopicStateChangeRecord, the MMM on the affected partition's leader drives the corresponding state transition that triggers a specific operation (create, start, stop, pause, resume, recover, delete).

MMM also maintains an Admin client connection to the source cluster. Every 60 seconds by default, it refreshes source metadata: Discovering new topics that match configured include/exclude patterns, syncing topic configurations, fetching consumer group offsets, and validating that the source cluster ID has not changed. That last check prevents silent data corruption if someone accidentally reconfigures the mirror to point at a different cluster.

ClusterMirrorCoordinator

ClusterMirrorCoordinator (CMC) handles state persistence. It follows the same coordinator pattern used by the group coordinator and the transaction coordinator, managing shards of an internal compacted topic called __mirror_state (defaults: compact cleanup policy, 50 partitions, replication factor 3). Each mirror partition's state is stored as a key-value record in this topic, with optimistic concurrency control through leader epoch and state epoch fencing.

MirrorFetcherThread

MirrorFetcherThread (MFT) does the heavy lifting. Extending Kafka's AbstractFetcherThread (the same base class used for intra-cluster replication), it fetches records from the source and appends them to local logs. Each thread maintains a dedicated NetworkClient with per-mirror authentication credentials, keeping SASL/SSL contexts isolated between mirrors. The fetcher manager keys threads by a three-dimensional identifier (fetcher ID, source broker endpoint, mirror name), enabling fine-grained load balancing and fast response to leader changes on the source.

Mirror partition lifecycle

A mirror partition progresses through a sequence of well-defined states. Figure 2 shows the full state machine.

• LOG_ALIGNMENT: The first step aligns the local log with the source. If no LME (Last Mirror Epoch) is found due to first time mirroring or unsupported source, the broker truncates to zero and mirrors from scratch, otherwise it only mirrors the delta. The LME record stores the last mirror offset and epoch from the previous session. The broker truncates records only at offsets up to the last mirror offset and with epochs up to the last mirror epoch, so any divergence is resolved.
• EPOCH_FENCING: The broker sends a BumpLeaderEpochs request to the controller, incrementing the local leader epoch by 10, with a re-bump threshold of 3. Without this guarantee, consumers on the destination could initialize with a committed epoch from the source that exceeds the local epoch, causing them to reject the leader as stale.
• MIRRORING: The fetcher thread begins pulling records from the source. Compressed batches are appended directly to the local log without recompression. The high watermark advances as local followers replicate the data. Group offsets are periodically synced from the source and clamped to the destination's valid offset range.
• ULE_RECOVERY: After mirroring starts, log divergence can only be caused by an ULE in the source cluster since only committed records are mirrored. When ULE support is enabled and the destination leader detects log divergence during a fetch, the partition transitions from MIRRORING to this state. The mirror fetcher is removed and the system waits for all replicas, not just ISR members, to converge to the truncated log end offset. Once every replica has caught up, the partition transitions back to MIRRORING and a new fetcher is created.
• PAUSING / PAUSED: An operator can pause mirroring without losing progress. Fetcher threads are torn down, but the partition remains read-only. Resuming transitions directly back to MIRRORING with fresh fetchers and no re-alignment.
• STOPPING / STOPPED: This is the failover path. The broker removes fetcher threads, records the LME for future failback, bumps the local epoch, appends ABORT markers for any in-flight transactions, and writes a MIRROR_PID_RESET control record to expire all producer state. The partition then becomes writable. Restarting a stopped mirror transitions back through LOG_ALIGNMENT, since the topic may have accumulated local writes while it was writable.
• FAILED: Errors trigger an automatic retry with exponential backoff (jittered), up to a configurable maximum number of attempts. Non-retryable errors require manual recovery (e.g. source cluster ID change, source topic deletion).

Data consistency

There are 2 mechanisms keeping the destination cluster's log consistent with the source: Log convergence handles epoch alignment and truncation, and transaction safety ensures uncommitted records don't leak after failover.

Log convergence

The destination leader epoch must always be greater than or equal to the source leader epoch (DLE >= SLE). Without this invariant, a consumer on the destination could initialize with a committed epoch from the source that exceeds the local epoch, refuse to fetch, and stall. The system maintains the gap through reactive bumps (when a fetched batch epoch approaches the local epoch), proactive bumps (when the gap shrinks below 3), and periodic bumps (during source metadata sync). Each bump increments the epoch by 10.

When mirroring starts or restarts, the local and source logs may have diverged. A 2-phase truncation protocol resolves this.

• The initial phase aligns leader epoch histories. The LME record stores the last mirror offset and epoch from the previous mirroring session. During LOG_ALIGNMENT, the broker truncates records only at offsets up to the last mirror offset, so any divergence is resolved. No LME means mirroring starts from scratch.
• The steady state phase aligns offsets within remaining epochs during normal fetch processing. The fetcher detects the source cluster's fetch version dynamically. If the source supports Fetch v12+, diverging epoch info in fetch responses allows inline truncation to the exact divergence point. If it does not, the fetcher disables truncation-on-fetch and falls back to an explicit OffsetsForLeaderEpoch request to find the truncation point. This handles both old and new source clusters, including live upgrades and downgrades.

Because the destination only fetches committed data during mirroring, any truncation after LOG_ALIGNMENT must originate from an ULE on the source. This applies regardless of fetch version. The mirror.unclean.leader.election.enable config gives operators a clear choice. When set to true, the partition transitions to ULE_RECOVERY and waits for all assigned replicas to converge before resuming, accepting data loss from the source ULE. When set to false (default), the partition moves to FAILED state, halting mirroring entirely.

Transaction safety

Cluster mirroring replicates with READ_UNCOMMITTED isolation, meaning uncommitted transaction records arrive at the destination before the source decides their fate. This is intentional: waiting for transaction completion would add latency and complexity to the data path.

During the stopping transition, the broker appends ABORT markers for any in-flight transactions, ensuring READ_COMMITTED consumers never see hanging transactions after failover. It also writes a MirrorPidResetRecord control record that expires all producer state entries in the ProducerStateManager. This allows new producers on the (now writable) destination to obtain fresh producer IDs without conflicts.

The PID reset mechanism propagates correctly through all replication topologies: Active-passive (A to B), failback chains (A to B to A), fan-out (A to B and A to C), and multi-hop chains (A to B to C).

Disaster recovery

Disaster recovery is the most natural application of cluster mirroring. Figure 3 shows the 3 phases.

Normal operation

Cluster A serves all production traffic. A mirror named a-to-b continuously replicates topics to Cluster B.

On Cluster B, create and start the mirror:

kafka-cluster-mirrors.sh --bootstrap-server B:9092 \
  --create --mirror a-to-b --mirror-config mirror.properties

kafka-cluster-mirrors.sh --bootstrap-server B:9092 \
  --start --mirror a-to-b --topics ".*"

The --describe flag lets you monitor replication progress at any time:

kafka-cluster-mirrors.sh --bootstrap-server B:9092 \
  --describe --mirror a-to-b

Failover

Cluster A goes down. A single command on Cluster B stops mirroring and makes all topics writable:

kafka-cluster-mirrors.sh --bootstrap-server B:9092 \
  --stop --mirror a-to-b

Producers and consumers switch their bootstrap servers to Cluster B. Because offsets are identical and consumer group offsets have been synced, applications resume from where they left off. No offset translation. No reprocessing.

Failback

When Cluster A recovers, the operator sets up a reverse mirror. The system detects that A was previously the source for these topics, looks up the LME recorded during failover, and performs incremental truncation rather than re-replicating the entire dataset.

On Cluster A, reverse mirror:

kafka-cluster-mirrors.sh --bootstrap-server A:9092 \
    --create --mirror b-to-a --mirror-config reverse.properties

kafka-cluster-mirrors.sh --bootstrap-server A:9092 \
    --start --mirror b-to-a --topics ".*"

When replication catches up, cut over:

kafka-cluster-mirrors.sh --bootstrap-server A:9092 \
  --stop --mirror b-to-a

The recovery point objective (RPO) depends on replication lag at the moment of failure. Cluster mirroring is asynchronous, so records produced on A but not yet replicated to B will be lost during an unplanned failover. For most DR use cases, this is an acceptable tradeoff: synchronous cross-cluster replication would impose latency penalties that compromise the primary cluster's performance. Watch out for KIP-1360, which plans to extend KIP-1279 with sync mode.

Cluster migration

Cluster mirroring supports source brokers as far back as Kafka 2.1, leveraging the client/broker forward compatibility introduced in Kafka 4.0. This opens a practical migration path that sidesteps the traditional step-by-step version upgrade entirely. Figure 4 shows the three phases.

The traditional migration path requires upgrading through every intermediate major version (2.x to 3.x to 4.x), performing the ZooKeeper-to-KRaft metadata migration in place, and hoping nothing breaks along the way. With cluster mirroring, the process becomes:

1. Stand up a new KRaft cluster running the target Kafka version.
2. Create a mirror from the old cluster to the new one.
3. Start mirroring. The destination automatically discovers topics, creates matching partitions (with identical topic IDs), syncs configurations, and begins replicating data.
4. Stop producers on the old cluster.
5. Monitor replication lag until it reaches 0.
6. Stop mirroring (topics on the new cluster become writable).
7. Redirect clients to the new cluster.
8. Decommission the old cluster.

On the new cluster:

kafka-cluster-mirrors.sh --bootstrap-server new:9092 \
    --create --mirror old-to-new --mirror-config old-cluster.properties

kafka-cluster-mirrors.sh --bootstrap-server new:9092 \
    --start --mirror old-to-new --topics ".*"

Monitor progress:

kafka-cluster-mirrors.sh --bootstrap-server new:9092 \
    --describe --mirror old-to-new

When lag reaches zero, cut over:

kafka-cluster-mirrors.sh --bootstrap-server new:9092 \
    --stop --mirror old-to-new

No intermediate version upgrades. No in-place metadata migration. No ZooKeeper-to-KRaft conversion scripts. The old cluster runs undisturbed until you are ready to shut it down.

Conclusion

Cluster mirroring represents a shift in how Apache Kafka handles cross-cluster data movement. By embedding replication into the broker, KIP-1279 eliminates an entire class of operational complexity while improving data fidelity: compressed batches pass through untouched, offsets remain identical across clusters, and consumer failover requires no translation.

For disaster recovery, a pair of commands replaces a multi-step runbook. For cluster migrations, it provides a clean upgrade path from ZooKeeper-based deployments running Kafka 2.1 all the way to modern KRaft clusters, with no intermediate version hops. Future work includes synchronous mirroring for zero-RPO requirements, tiered storage integration, and support for diskless topics.

Your data wants to be free, so if you have been putting off a cluster migration because of the risk, or running DR drills that involve a binder of runbooks and a generous maintenance window, Cluster mirroring offers a simpler path, so give it a try. Happy hacking.