I wanted to blog about this for a very long time, but because of work and being too lazy to turn on my laptop on the weekends, I now finally have a couple of hours to sit down and start blogging.
"Integration is DEAD."
"No, no, let's talk about microservice!"
"What is that again? Integration, you mean SOA? It's lame...".
During my recent conversations with many people about Integration, these comments came up a lot. In fact, if you happen to attend any the software conferences, you see the rooms are full when the topic is about "containers" and "microservices." I understand the nature of a developer, always seeking for the latest and greatest technology that's out there. BUT! In my opinion, I don't think Integration is going AWAY if you containerize your application. Neither will it DISAPPEAR in a microservice architecture. In fact, Integration just gets more complicated and tricky when you try to break an application into smaller pieces and have them running all over the place in a distributed environment!
It was hard to deal with when we have to orchestrate, choreograph, and compose all the coarse-grained services together, and imagine letting tinnier, smaller and more refined services talk to each other. Yeah, trying to get an oversized heavyweight strong bull to co-ordinate and communicate is one thing. 
I don't think to have to conduct thousands of small cute ducklings, (I know, they are now smaller to handle one by one, light and easy to replace) is going be any easier. :-)
Maybe the word Integration was abusively used by too many vendors (Us too), and it is too often associated with ESB, which IMHO should be one of the Integration strategies. ESB can be overly complicated with wrapped around RPCs. Integration is about connecting, composing between services, neglecting integration when implementing microservice architecture (MSA) is going to be disastrous. I am sure a lot of smart people out there designing the MSA have already taken this into account. But they probably did not realize that because Integration is already deeply ingrained into their minds, even in microservice we still want loose coupling. Again, in my opinion, the wall between application development and integration are getting thinner and thinner, by that I mean you do integration in a microservice way and apply integration in your microservice. So, yes, integration needs to change, adapt and adjust accordingly but it never goes away:
- Current deployment method
- As services are updated more frequently than ever before, we need to quickly respond to the change, in the integration layer. We can package it into the container and standardize the deployment method across the company.
- Speed and automated delivery
- A quicker response to a developer, continuous integration and delivery, and faster time to markets can be achieved by adopting the DevOps trend. Integration should also be part of this cycle. Not OUTSIDE!
- Flexible for changes
- Making sure the message can be routed to the correct service, allowing automatic discovery of services, with load balance capability that shares the load no matter its synchronized or event base call. Providing and composed with flexible contracts like APIs between service can also minimize the impact on the changes.
- More robust towards failure
- In the integration, it's all about the isolation of failure, and the handling of errors, and increase service resiliency. Also, the ability for it to adopt version changes, and service lifecycle management is also one of the important factors in Integration.
Some concepts of doing integration in MSA compared with the monolithic application are poles apart. For example, continue from the bull and duckling theme. When you try to get a couple of stubborn, massive bulls together and adjust position it takes a lot of energy, and often you will need to be creative and smart to improve according to their position. Therefore, the pipelines have to be smart and do many manipulations. Whereas the ducklings are easy to move around and cheap to replace. If one got sick, naughty or does not have the right color you need, simply toss it and get a newly hatched egg. (Alright, I don't mean any animal cruelty here; it's just a metaphor. ). The pipe that brings these microservice can be dumb or simple. Because each duckling is cheap to create.
In my eye, microservice should never go freely to create on wherever and whenever it should be at least logically bound. I call this, application domain. By circling these microservices, we will be able to see a much clearer picture of inbound microservices integration and outbound microservices integration.
The inbound microservices integration will be a lot more flexible in protocols, data formats because of the possibility to connect to various datasources and SaaS application. Since they are bounded within the same application domain, it's likely the data models are shared between these microservices. They tend to be more coupled with each other (although service autonomy is where we are trying to get. They are probably handled by the same person.) There will be a couple of microservices responsible for composing some of the other microservices together too. The logic inside should be simple and straightforward. And each of the microservices should easily take care of the protocol and data extract itself.
On the other hand, the outbound microservices integration should be strictly loosely coupled with inbound microservices. Preferably access commonly managed endpoints from each application domain, yes, I mean APIs. It is mostly device or client bound. As it often requires putting together a more coarse-grain service like the old ESB days for the client application or from a process.
Both inbound and outbound integration can be synchronous and asynchronous; it all depends on the need of your system. So, don't limit yourself to just one option. I strongly recommend the Enterprise Integration Pattern Book, as I said, the integration problem is still there. The patterns are extremely useful in this manner.
Agile Integration is Red Hat's opinionated way of doing the integration. And most importantly, it's backed up by Fuse (Apache Camel) and OpenShift.
It talks about 3 main pillars, which resonate my integration beliefs in the MSA world.
- Distributed Integration
- Lightweight, support distributed deployment
- Pattern Based Integration
- Reusable Connectors
- Containers
- Cloud native solutions
- Lean artifacts, individually deployable
- Container based scaling and high availability
- API
- Well defined, reusable, and well-managed end-points
- Ecosystem leverage
So! I hope my murmur on this blog post will at least make you think "Integration is NOT LAME." And start thinking about it the next time you create your next microservice.
For a framework for building enterprise Java microservices visit WildFly Swarm and download the cheat sheet.
Last updated: August 29, 2023