The term asynchronous means "not occurring at the same time." In the context of distributed systems and messaging, this term implies that request processing will occur at an arbitrary point in time. Asynchronous interactions hold many advantages over synchronous ones, but they also introduce new challenges. In this article, we will focus on specific considerations for choosing the asynchronous-messaging infrastructure for your event-driven systems.
We will start by looking at the subtle differences between asynchronous interaction styles based on the business value and semantic type of the messages being delivered. Considering these differences helps us to identify messaging patterns, which we can use to determine the kind of messaging system that we need.
Note: At Red Hat, we love any open source technology, so I use three open source messaging frameworks—Apache Qpid, Apache ActiveMQ Artemis, and Apache Kafka—for my examples. Red Hat AMQ is our flexible messaging platform that includes all three frameworks, making it easy to choose the right tool for your needs.
The business value of different message types
Not all messages are created equal. Some are valid and valuable only for a short period and become obsolete later. Some are valuable until they are consumed, regardless of how much time has passed. And some messages are valid and useful for repeated consumption. By considering the validity and value of messages relative to time and consumption rate, we can qualify interaction styles between services into three categories, as described in Figure 1.
Let's consider each of these categories.
Volatile
Messages in a volatile messaging system are ephemeral, and the value of the message is time-bound: They are valuable now, but soon they will not be. There is no point in storing events that will soon be useless. For this type of event, a volatile messaging system yields the best performance with the lowest possible latency as writing to the disk is skipped. In this type of scenario, the messaging system is aware of consumers and disseminates events to all consumers who are online at the time of publication. When consumers disconnect from the system, the messaging system forgets about them. This type of system is vital for its ability to handle a large number of dynamic clients that have low-latency interaction needs, such as Internet-of-Things (IoT) devices.
Durable
This more traditional type of message system knows about registered consumers and stores messages durably until every registered consumer has read them. This type of system works well for scenarios where the consumer might be disconnected when the event is published. The system holds on to the messages until every consumer has re-connected and consumed the relevant events. Once an event has been fully consumed, the message broker discards the messages. The goal is to offer reliable messaging among services with strong ordering and delivery guarantees.
Replayable
Here, the messaging system is not aware of consumers or event registrations. It merely stores events and publishes them to a stream for a given period, or until capacity is reached. In this type of system, an existing or new consumer can join at any time, connect and consume events, and even replay the stream from the beginning. Consumers can move back and forth in the stream as required. The driving force for this type of messaging system is scalability, combined with the ability to replay messages.
Message semantics
Apart from the technical characteristics of messages, it is essential to distinguish the language that we use—i.e., the semantic aspects—and the intent of the interactions. Some messages are targeted for a specific consumer and demand concrete actions. Some query the latest state of a system without requiring a state change, and some notify the world about a change that has happened in the source system. From a messaging-semantics perspective, there are three message types, as described in Figure 2.
Let's consider each of these message types.
Command
A command is a request for action that usually leads to a state change on a known target system. Typically, a response indicates that the action was completed, and there might even be a result associated with the response. When a response is expected, commands are typically implemented over synchronous protocols such as HTTP. It is also possible to apply request-and-response or fire-and-forget command styles over asynchronous messaging systems. Command-based asynchronous messages require coupling between the source system and the target system, in the form of command semantics.
Query
A query is like a command, but it is a read-only interaction that does not lead to a state change. By nature, a query expects a response. It is common to see synchronous implementations of this message type, but asynchronous and non-blocking implementations over messaging systems are also typical. Even fire-and-forget interactions for long-running operations, where a response is written to a different location, are not unusual.
Event
An event is a notification that something has changed. A system sends event notifications to notify other systems of a change in its domain. An event is different from a command in that often the event-emitting system doesn't expect a response. In addition to being asynchronous, event messages are not targeted to a specific recipient, which makes further decoupling possible. Similar to other asynchronous interactions, events are implemented as messages on queues, which are often called streams.
Note: See Martin Fowler's presentation, "The many meanings of event-driven architecture," for in-depth coverage of the different types of events in a messaging system.
Choosing a messaging system
The Law of the instrument approach defined by Abraham Maslow says, "If the only tool you have is a hammer, treat everything as if it were a nail." Following that approach, you could certainly use a classic message broker such as Apache ActiveMQ Artemis to implement the different interaction styles described in this article. The technology is familiar to many, which would make it easy to use from the start. On the other hand, it would be challenging to develop use cases such as replayable messaging with a classic message broker.
On the other end of the extreme, you could try using something like Apache Kafka for every messaging scenario. Kafka would require a more substantial amount of hardware resources and human effort to manage, but use cases requiring replayable messaging or extreme scalability would be covered.
While both of the above approaches are fine in some cases, when you have a large number of services with different messaging needs, using the right tool for the right job is a better option. Mapping the previously described messaging patterns is a useful tool for deciding what messaging infrastructure you need. In Figure 3, I have mapped the characteristics of three different types of message broker system to the different messaging scenarios and types of messages we've discussed.
Each of the three frameworks in Figure 3 is effective for a different type of messaging scenario and need. As I mentioned at the beginning of this article, Red Hat AMQ packages all three of them—Apache Qpid, Apache ActiveMQ Artemis, and Apache Kafka—so that you can choose the right tool for the right job.
Conclusion
There are many aspects to consider when choosing the right event-messaging infrastructure for your needs. I hope that the considerations and the mapping tool presented in this article will help you get one step closer to deciding.
Last updated: July 28, 2020