After a couple of months in Developer Preview, the Debezium Apache Kafka connectors for change data capture (CDC) are now available as a Technical Preview as part of the Q4 release of Red Hat Integration. Technology Preview features provide early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process.
Red Hat Integration provides Debezium connectors for capturing changes from the following databases:
- MySQL Connector
- PostgreSQL Connector
- MongoDB Connector
- SQL Server Connector
Debezium connectors are based on the popular Apache Kafka Connect API and are suitable to be deployed along Red Hat AMQ Streams Kafka clusters.
AMQ Streams is a Red Hat Integration component that provides Red Hat’s distribution of Apache Kafka and the popular CNCF sandbox project Strimzi. AMQ Streams makes running and managing Kafka an OpenShift-native experience by delivering OpenShift Operators, which provide a simplified and automated way to deploy, manage, upgrade, and configure a Kafka ecosystem on OpenShift.
With this addition, Red Hat Integration now makes available more components to connect systems along the whole enterprise ecosystem. Along with Apache Camel’s 200 connectors, users can connect to practically everything—from legacy systems to Software-as-a-service (SaaS) applications, and application programming interfaces (APIs) to Internet of Things (IoT) devices.
What is Change Data Capture (CDC)?
Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access the change events in the order in which they occurred.
What is Debezium?
Debezium is a set of distributed services that captures row-level changes in databases so that applications can see and respond to those changes. Debezium connectors record all events to a Red Hat AMQ Streams Kafka cluster, and applications consume those events through AMQ Streams.
CDC in action with Debezium
You can check Sadhana Nandakumar’s blog post in which she explains how to make use of Red Hat Integration to create a complete CDC pipeline. In the example she captures the changes as they occur using Debezium and streams them using Red Hat AMQ Streams. Then, she filters and transforms the data using Red Hat Fuse and sends it to Elasticsearch, where the data can be further analyzed or used by downstream systems.
You can download the Red Hat Integration Debezium CDC Technical Preview connectors from the Red Hat Developer site. If you have requests or questions related to running the Debezium Technical Preview, please let us know by sending an email to the debezium-cdc-preview mailing list.
Last updated: July 1, 2020