Milano JUG

Our journey to the data mesh: Boosting data in a microservice world

Abstract:

What is the best data architecture in the age of microservices and “you-build-it-you-run-it” teams? How should microservices share data between themselves to minimize coupling? How should microservices share data with offline data consumers such as BI, CRM, or ML training pipelines? How can we reconcile data quality, trustworthiness, and single-source-of-truth with autonomous teams releasing multiple times per day? Are centralized data governance teams as proposed by data lake architectures compatible with microservices?

Data meshes have been proposed as a new data architecture paradigm to answer the questions above, but there are very few public reports on actual implementations and their challenges. We will share the data mesh journey that we started two years ago in eDreams, one of the largest online travel agencies in the world. We will focus on three topics: motivation, technical design, and transformational approach.

First, we'll explain why we think data meshes are a better fit for microservice architectures. Second, we will show the architecture of our current implementation, including how the mesh is connected to the online platform, the technologies we used, and the trade-offs we made for team autonomy, data ownership, quality and accountability, PCI and privacy compliance, self-service data access and reporting, infrastructure cost and budgeting. Third, we will share the organizational challenges we faced when replacing the multiple data systems that were working well at the local level. Finally, we will conclude with our current list of open issues and future work.

About the speaker: Carlos Saona-Vázquez

Carlos is the Chief Architect at eDreams Odigeo, the largest flight travel agency in Europe, where he has led architectural transformations, such as the migration to microservices and the adoption of chaos engineering or the creation of a data mesh. Before eDreams, Carlos served in multiple technical roles designing distributed, scalable and fault-tolerant systems for health care, computer security, and video game companies.

Event details

Date: Feb 23, 2023 18:30 

Where: L'incontro si terrà Giovedì 23 Febbraio 2023 in Via Gustavo Fara, 26, 20124 Milano MI (mappa) presso eDreams Via Fara 26 20124 Milano - evento gratuito previa registrazione OBBLIGATORIA (vedi dettagli) grazie alla co-organizzazione con eDreams e Red Hat.