Change data capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
Thus, change data capture helps to bridge traditional data stores and new cloud-native event-driven architectures. Meanwhile, Debezium is a set of distributed services that captures row-level changes in databases so that applications can see and respond to those changes. This general availability (GA) release from Red Hat Integration includes the following Debezium connectors for Apache Kafka: MySQL Connector, PostgreSQL Connector, MongoDB Connector, and SQL Server Connector.
CDC and event-driven microservices with Debezium
Debezium's log-based change data capture has the advantage of capturing all of the data changes that were registered in the database, providing a trustable source. There is no delay compared with querying the database or overhead. Debezium also provides a transparent mechanism for applications and data models, avoiding the need to pollute current systems design. Reading for the transaction log also offers a low overhead with no risk of missing any events compared with polling.
Based on the popular Apache Kafka Connect API, Debezium Apache Kafka connectors are suitable for deploying with Red Hat AMQ Streams Kafka clusters. They record all events to a Red Hat AMQ Streams Kafka cluster, and applications consume those events through AMQ Streams.
Debezium also allows the capture of delete events, as well as information about the old record state and other metadata that can then be shared as part of the event for further processing. Debezium change event structure includes information about the table's key along with information regarding the value with the previous state, current changes, and metadata info. These events can be serialized as the familiar JSON or Avro formats, while support for CloudEvents is coming in a future release.
Use cases where change data capture is especially useful include:
- Data replication to other databases in order to feed data to other teams, or as streams for analytics, data lakes, or data warehouses.
- Microservices data exchange to propagate data between different services without coupling, keeping optimized views locally, or for monolith-to-microservices evolutions.
- Auditing, cache invalidation, indexing for full-text search, updating CQRS read models, and more.
Get started with CDC Debezium Apache Kafka connectors
Debezium Apache Kafka connectors are available through Red Hat Integration, which offers a comprehensive set of integration and messaging technologies that connect applications and data across hybrid infrastructures. This agile, distributed, containerized, and API-centric solution provides service composition and orchestration, application connectivity and data transformation, real-time message streaming, change data capture, and API management—all combined with a cloud-native platform and toolchain to support the full spectrum of modern application development.
This Red Hat Integration release provides full support for Debezium connectors to capture changes from the MySQL Connector and PostgreSQL Connector. Connectors for MongoDB and SQL Server are now included and delivered as Technical Previews. Also known as Red Hat Integration CDC connectors, Debezium provides access to the benefits of open source for the enterprise—like community-driven upstream innovation—delivered with enterprise-level support to help your organization safely use open source technology. Check the complete supported configurations for more information.
Get started by downloading the Red Hat Integration Debezium CDC connectors from the Red Hat Customer Portal. Also, don’t forget to check out Gunnar Morling’s webinar on Debezium and Kafka from the DevNation Tech Talks series, or his talk at QCon.
Last updated: November 8, 2023