This article introduces the new Debezium Db2 connector for change data capture, now available as a technical preview from Red Hat Integration. Get a quick overview of using Debezium in a Red Hat AMQ Streams Kafka cluster, then find out how to use the new Db2 connector to capture row-level changes in your Db2 database tables.
Note: Change data capture, or CDC, is a well-established software design pattern for monitoring and capturing data changes in a database. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read the change-event streams and access change events in the order that they happened.
What is Debezium?
Debezium is a set of distributed services that captures row-level database changes so that applications can see and respond to them. Debezium connectors record all events to a Red Hat AMQ Streams Kafka cluster. Applications use AMQ Streams to consume change events.
AMQ Streams is a Red Hat Integration component that provides Red Hat’s Apache Kafka distribution and the popular Cloud Native Computing Foundation sandbox project Strimzi. AMQ Streams makes running and managing Kafka a native experience in Red Hat OpenShift. AMQ Streams Operators provide a simplified and automated way to deploy, manage, upgrade, and configure a Kafka ecosystem on OpenShift.
The Debezium connector for Db2
Debezium's Db2 connector was inspired by Debezium's implementation of SQL Server. This connector is based on the Abstract Syntax Notation (ASN) capture and apply agents used for SQL replication in Db2. Db2 connector agents generate change data for tables that are in capture mode. They also monitor the tables and store change events for table updates. The Debezium connector then uses an SQL interface to query change-data tables for change events.
How the Db2 connector works
The Db2 connector uses the ASN libraries that come as a standard part of Db2 for Linux, Unix, and Windows (Db2 LUW). You can manually add the ASN libraries to Db2 zOS. To use ASN and the Db2 connector, you must have a license for the IBM InfoSphere Data Replication (IIDR) product. You do not, however, need to have IIDR installed. The Db2 connector has been tested with Db2/Linux 11.5.0.0.
The Debezium Db2 connector generates a data-change event for each row-level INSERT, UPDATE, and DELETE operation. Each event contains a key and a value. The structure of the key and the value depends on the table that was changed.
Before running a Debezium Db2 connector to capture changes committed to a Db2 database, a database administrator must put tables into capture mode. Debezium provides a set of user-defined functions that you can use for this purpose.
Installing Db2 connector
To install Debezium's Db2 connector, follow the procedures in the documentation Installing Debezium on OpenShift. Unfortunately, the JDBC driver for IBM Db2 is not supplied as part of the Debezium connector due to licensing requirements. You can manually download and copy the JDBC driver as part of the provided container image.
When the connector starts, it takes a consistent snapshot of the Db2 database tables that the connector is configured to monitor. The connector then generates data-change events for row-level operations and streams change-event records to Kafka topics. Figure 1 shows a Db2 connector configuration.
Get started with Debezium Apache Kafka connectors
Debezium Apache Kafka connectors are available through Red Hat Integration, which offers a comprehensive set of integration and messaging technologies that connect applications and data across hybrid infrastructures. This agile, distributed, containerized, and API-centric solution provides service composition and orchestration, application connectivity and data transformation, real-time message streaming, change-data capture, and API management—all combined with a cloud-native platform and toolchain to support the full spectrum of modern application development.
Get started by downloading the Red Hat Integration Debezium connectors from the Red Hat Developers Portal. You can also check out Gunnar Morling’s webinar on Debezium and Kafka (February 2019) from the DevNation Tech Talks series or his more recent presentation at QCon (January 2020).
Last updated: November 4, 2020