Integration is still around but in a different form. So, what does modern integration look like? Looking at how agile scrum has taken over traditional waterfall development framework, by enabling shorter delivery cycles, faster feedback, and having the flexibility to rapidly adapt to changes. I believe it’s time for traditional integration to be agile again. By breaking up traditional ESB into distributed microservices.
A little recap of what should be in Agile integration:
- Distributed Integration
- Lightweight, support distributed deployment
- Pattern Based Integration
- Reusable Connectors
- Microservices
- Containers
- Cloud native solutions
- Lean artifacts, individually deployable
- Container based scaling and high availability
- API
- Well defined, reusable, and well-managed end-points
- Ecosystem leverage
Based on these three principles, I was asked to create a reference architecture for it. After giving it some thought, drawing up numerous boxes and diagrams, something clicked when I watched a 3-year-old playing. And here, this is my answer…
Yup. That is it! Reference architecture of Agile Integration. Architecture is often needed during the design phase of a software project; it provides a structure and backbone for the software project. Once it’s done, it’s done; any architectural modification will be difficult and can bring disastrous results. We all knew “one thing will never change in software development is Change”. Maybe it’s because of the regulation requirement, market demands or simply learning more about the business domain. So I started to think, can we build an architecture that is flexible for change, and can be shaped into the needs of a project as needed? So based on that concept, this is my reference architecture for Agile Integration, the modern integration application development that enables flexibility from many angles.
There are the major components in my reference architecture diagram for Agile Integration:
PaaS - Provides a foundation for fast pace software development by allowing the developer to self-service, as well as an enable-automated provision. It will also boost operational efficiencies by making the entire environment DevOps ready.
Security/Identity management (IAM) - Handles basic authentication and authorization for application interfaces and platforms.
Automation - Automates both the processes of building applications and rolling deployment strategies for upgrades. Together with continuous integration tools, it can achieve continuous delivery of the application software.
Logging and Tracing - What was supposed to be easy in a monolithic world, has now become one of the challenges in distributed microservice architecture for integration. As it has now become harder to have a unified view of how things are doing. It is important to have a way to see logging as a whole. And has the tracing capability for different activities.
Containerized packages - Container technology allows us to build immutable portable lightweight application package, that is language agnostic. The configuration is independent of this build allowing the same package to deploy quickly in multiple environments.
Container management - In a large-scale system, it is essential to have a convenient management platform to manage all the containers running the applications. Taking care of the monitoring, discovery, recovery, and failover of the running containers.
Microservices - Large applications/services are broken down into easy maintainable pieces and can be independently developed and deployed in a distributed environment. Each of them should be built with failure in mind.
Load Balance/Service discovery/Network management - Microservices are built for flexibility, according to current system load, the load should be automatically balanced for each running instance. All services should be registered and can be found in the system without complex setup. Ideally, a proxy endpoint for external users to hide away the deployment complexity.
Layered architecture - Logically organize microservice into a set of layers, each has its own responsibility to avoid duplication of effort and make it easier to replace later on. Currently, there are 4 layers:
- Gateway layer - provides simple gateway routing capability such as versioning, and dealing with a different platform of devices.
- Composite layer - an important middle tier that handles composition of multiple microservices. They do more complex routing from processing the content data itself and sometimes handle more complex data aggregation or normalization.
- Base layer - like the name, which is most likely, represents the basic components of the system. Handles data retrieval or business logic processing.
- Anti-corruption layer - This layer handles interface to a legacy application or anything that works against microservices quick and flexible principle. This layer is built in the protection wall to your system, by doing the transforming job and translation between two very distinct implementations of the system.
Application domains - This one is more like a pattern than architecture. But since we mention layers in software I also wanted to touch a little bit on this. Because it’s all about reining microservices into a more organized way. It’s a good practice to have each application domain defined (using the same set of the data model). And have other external integration microservices to wire between each domain. One other good thing about microservices is that you have the flexibility to move each small microservice to different domains where it will better fit.
API management - Managing the APIs by reinforced access policy, collecting statistics around the usage of APIs. Also building a healthy eco-system among the developer and users of APIs. Further to allow direct revenue source from the APIs.
There is certainly some limitation in today’s technology to allow a complete pliable architectural form. But I believe this model can bring flexibility into the system without significant impact to the system when changes occur. I will provide more details in my upcoming posts.
Take advantage of your Red Hat Developers membership and download RHEL today at no cost.