Consistent access and delivery with Data Integration

Data consolidation involves designing and implementing a data integration process that feeds a datastore with complete, enriched data. This approach allows for data restructuring, a reconciliation process, thorough cleansing, and additional steps for aggregation and further enrichment.

Extract, transform, load (ETL)

ETL offloaded the transformation of raw data to usable data from the target datastore. This transformation process can become a bottleneck. In cloud computing, there's no added benefit, such as the reduction in target server loads.

Extract, load transform (ELT)

ELT  is highly scalable–store as much data as you need in its raw form and get it to the target quickly. No specialized infrastructure for transformation processes is necessary before it lands at its destination.

Data federation uses a pull approach where data is retrieved from the underlying source systems on-demand. This pattern provides real-time access to data. Data federation creates a virtualized view of the data with no data replication or moving of the source system data.

Composite service

A composite service implements the aggregator pattern. It combines the data from different, distinct services in a meaningful way and serves this response to the consuming application.

Data virtualization or Enterprise Information Integration (EII)

Data virtualization (Ell) cpmbines large sets of diverse data sources in a way that makes them appear to a data consumer as a single, uniform data source. It uses data abstraction to provide a common data access layer.

Data propagation involves the promotion of data updates on two levels. At the application level, an event in the source application triggers processing in one or more target applications.  At the datastore level, an event in the source system triggers updates in the source datastore. These change events are then replicated in near real-time to one or more target datastores.

Enterprise Application Integration (EAI)

EAI is distributed, lightweight, and scalable for elastic operating environments—the integration itself may be deployed as a containerized application.

Enterprise Data Replication (EDR)

In distributed and microservices architectures, replication allows applications to be more reliable. 

The data service needs may be replicated and colocated with it and stored in a manner that is more usable by that particular service. This reduces overhead and latency.

Event sourcing

Event sourcing is a pattern that makes sure that all changes to an application’s state are stored as a sequence of events. These events can then be used for temporal queries allowing for the reconstruction of past states and activity replay. This pattern is useful for creating audit logs, debugging, and use cases that require the reconstruction of the state at a specific point.

Read more

Streaming data and event stream processing

ESP involves taking action on a series of data points that originate from a system that continuously creates data. In this context, an event is a data point in the system and the stream is the continuous delivery of those events. This series of events is also referred to as streaming data. The types of actions that are taken as a result of these events include aggregations, analytics, transformations, enrichment, and ingestion into another datastore.

Start tutorial

Distributed caching and in-memory data grids

The concept of caching is to provide storage capacity for data on a system that's used to serve future requests more quickly. Data that's stored in cache is placed there because it's frequently accessed or contains duplicated copies of data stored in another datastore. The overarching goal of caching is to improve performance.

Read more

Data replication

CDC can be used for data replication to multiple databases, data lakes, or data warehouses, to ensure each resource has the latest version of the data. In this way, CDC can provide multiple distributed and even siloed teams with access to the same up-to-date data.

   Click here to see a diagram

Auditing

Facing today's strict data compliance requirements, and heavy penalties for noncompliance, it is essential to save a history of changes made to your data. CDC can be used to save data changes for auditing or archiving requirements.

   Click here to see a diagram

Microservice data exchange

CDC can be used to sync microservices with monolithic applications, enabling the seamless transfer of data changes from legacy systems to microservices-based applications.

   Click here to see a diagram

Mono-to-micro Strangler Pattern

Through an incremental approach, you can take scoped components and move them to a new microservices architecture. Use CDC to stream changes from the monolithic database over to the microservices database and the other way around.

   Click here to see a diagram