With the release of Red Hat AMQ Streams 1.2, Red Hat Integration now includes a developer preview of Change Data Capture (CDC) capabilities to enable data integration for modern cloud-native microservices-based applications. CDC features are based on the upstream project Debezium and are natively integrated with Apache Kafka and Strimzi to run on top of Red Hat OpenShift Container Platform, the enterprise Kubernetes, as part of the AMQ Streams release.
What is Change Data Capture (CDC)?
Change Data Capture, or CDC, is an old software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. Often used in data-warehousing solutions to react to OLTP data changes, CDC has had a rebirth in the microservices architecture as part of a distributed data integration approach.
When running in a container environment, CDC captures row-level changes to a database table and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access the change events in the order in which they occurred.
The primary use of CDC is to enable applications to respond almost immediately whenever data in databases change. Applications can do anything with the insert, update, and delete events, including:
- Streamlining data integration and data replication
- Cache invalidation and feeding full-text search indexes
- Extracting microservices out of simplifying monolithic applications
What is Debezium?
Debezium is a set of distributed services that captures row-level changes in your databases so that your applications can see and respond to those changes. Debezium records all row-level changes committed to each database table in a transaction log. Applications simply read the transaction logs they're interested in and see all of the events in the order in which they occurred. Debezium is durable and fast, so apps can respond quickly and never miss an event, even when things go wrong.
Debezium provides connectors for monitoring the following databases:
Debezium connectors record all events to a Red Hat AMQ streams Kafka cluster, and applications consume those events through AMQ streams. Debezium uses the Apache Kafka Connect framework making all of Debezium’s connectors, Kafka Connector source connectors, and, as such, they can be deployed and managed using the AMQ streams Kafka Connect resources.
Red Hat Integration
Red Hat Integration’s Debezium CDC Apache Kafka connectors are available as a developer preview today as part of Red Hat integration. For customers embracing a cloud-native development model, Red Hat offers a full portfolio of technologies that enable them to build microservices-based applications, exchange information synchronously or asynchronously between them, integrate them with legacy systems, and run them on an industry-leading enterprise container and Kubernetes application platform.
In case of any requests or questions related to running the Debezium Developer Preview with Red Hat AMQ Streams on OpenShift, please let us know by sending an email to debezium-cdc-preview mailing list. For more information on Debezium and AMQ streams, see https://debezium.io/docs/amq-streams/
Last updated: September 3, 2019