Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Data as a microservice: Distributed data-focused integration

May 23, 2019
Hugo Guerrero
Related topics:
Microservices
Related products:
Developer Tools

Share:

    Microservices is the architecture design favored in new software projects; however, getting the most from this type of approach requires overcoming several previous requirements. As the evolution from a monolithic to a distributed system takes place not only in the application space but also at the data store, managing your data becomes one of the hardest challenges. This article examines some of the considerations for implementing data as a service.

    When following microservices design guidelines, we find references to data handling at several points. Some of those common directions include:

    • Use private databases per service for loose coupling.
    • Embrace eventual consistency.
    • Implement saga pattern for eventual consistency transactions.
    • Use Command Query Responsibility Segregation (CQRS) and API composition.

    With these points in mind, when targeting loose coupling as an essential piece in the architecture, shared databases now become an antipattern, making transactions and querying increase in difficulty. The use of datastores per service requires encapsulated data, whereas the interaction with other domains of the architecture should happen only at the API level, encouraging us to hide data implementation details. As a result, using lightweight frameworks like Spring Boot is just one of the first steps in the microservices journey.

    The query challenge

    Because we have data store per service, we need to make it available for consumption by other services becoming the entry point for this domain. As all the data calls happen at the service level and according to their domain, when a view of composite data is required, a traditional “table join” is no longer an option like we used to have in the shared database implementation. Additionally, we can not write a service that queries the private data stores and aggregates the data because it violates the encapsulation design.

    To solve the previous challenge, we need to bring back to the microservices architecture the use of well-established Enterprise Integration Patterns (EIP) like content enrichment and aggregator. Most of the time, these patterns have been rebranded as API composition pattern and commonly implemented in components like the API gateway.

    In general, the API composition pattern involves the addition of another service, which makes the calls to the underlying data services with the required information to compose the required data view. As seen in the image below, it will first query the Customer Service for the basic information and then use that information to retrieve the history for this customer from the Payment Service.

    At first glance, this looks like a simple composition task; however, the business typically requires more and more control of how the data is consumed and more logic is added to this type of services. Restrictions on the amount of data to be retrieved, permissions of the consuming end user, etc., are common business requirements, making the implementation of this type of service a full-time maintenance task.

    On the other hand, the Command Query Responsibility Segregation (CQRS) tries to tackle the query challenge focusing on maintaining one or more materialized sets of data aggregated from multiple source events. As a result, the complexity of the system increases as an event bus is now a requirement. We will talk about this pattern in future articles.

    Distributed data integration

    As has been discussed before, the distributed nature of microservices makes service-to-service communications and service composition vital for a successful implementation. Trying to implement every service from the ground up in an ad hoc fashion, even though possible, is not always recommended, especially when specialized tools exist already and help us simplify the work.

    In fact, making data available for external consumption through services is one example where coding everything from scratch is avoidable. We can expose data resting in different stores, not only relational databases using existing frameworks that help us implement the API composition pattern but also simple data-as-a-microservice services.

    A distributed data-focused integration provides integrated access to data in any of its store implementations through a single uniform API. Data integration allows joining and unioning data even if the information resides in more than one type of sources beyond SQL or JDBC. Contrary to a database management system, it should not store any data but act as a “single point” interface to optimally access data sources.

    Markedly, this type of framework should be compatible with the distributed nature of microservices. Hence, the engine and implementation should be lightweight and be able to be deployed as a container in cloud environments. Correspondingly, you should have the flexibility to execute standalone the components in runtimes like Spring Boot or embed it in your application.

    Summary

    All things considered, going beyond building services in a microservice architecture involves the use of generally established practices like Enterprise Integration Patterns and Data Integration techniques. The ability to query different data sources and join them to expose meaningful information while providing a secure access layer in a lightweight and distributed way, can simplify your applications. And, as Christian Posta said in his article, "data, data integration, data boundaries, enterprise usage patterns, distributed systems theory, timing, etc, are all the hard parts of microservices (since microservices is really just distributed systems!).”

    Last updated: February 6, 2024

    Recent Posts

    • How to use RHEL 10 as a WSL Podman machine

    • MINC: Fast, local Kubernetes with Podman Desktop & MicroShift

    • How to stay informed with Red Hat status notifications

    • Getting started with RHEL on WSL

    • llm-d: Kubernetes-native distributed inferencing

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue