Messaging is a critical aspect of integrating systems, and while there are many different messaging platforms and infrastructures, a common request is for "zero loss of messages."  From there, the terms "Persistence" and "Durability" often get thrown around, but what do those two things really mean?

Persistence

At a basic level, persistence means that when failure occurs during message processing, the message will still be there (where you found it the first time) to process again once the failure is resolved.

Take JBoss Active MQ for example. In AMQ we have brokers that do the communicating of the messages. For simplicity's sake, let's assume we only have a single broker doing the communication to and from a queue. Should this broker be shut down while a message is in the queue, ready to be processed, once the broker comes back up the message will be processed normally.

So how does this work?  In order for messages to "persist" they must be stored somewhere other than just broker memory.  Depending on the platform this could be a temporary folder, a database, a log file, etc.

Now, why would anyone not use persistent messaging?  Well, for one thing it tends slows things down. Otherwise, maybe some messages are okay to lose in the event of a broker shutting down, and it's not worth the complexity.

If you think of messaging in the context of status checking, the system may want to periodically ensure that a device is up and running. The device send a status message every few minutes. In the event of a broker restart without using persistent messaging we may lose 1 status message, but that might not be a problem since another message is probably on its way --- in this case some data-loss may be acceptable.

Durability

Queues and Topics are important parts of messaging (particularly JMS). A queue by itself is great for point-to-point messaging, often one producer to one consumer. Topics on the other hand are most often used when you have a single producer (or multiple for the same purpose) and many consumers.

A common pattern is to have the producer send the message to the topic and then have the queues subscribe to the topic. This allows each queue to receive its own copy of the message. But what happens to the message if it is sent to the topic, but no queues are online (remember our queues subscribe to the topic)?

This is where durability comes into play. When a durable subscription is set up between a queue and a topic, the queue can be offline when the message hits the topic. Once the queue comes back online, the message can be received.

If the subscription is non-durable, then any messages received to the topic while the topic subscriber is offline will not be received by the subscriber (in this case the queue).

Preventing Message Loss

So what do we need to prevent message loss?  If you are using both queues and topics, then using both persistent messaging and durable subscriptions is your best bet. This will ensure you have a back up of the message in case of broker failure and that your subscriptions will always receive the proper messages. Just remember that certain messaging systems, such as Amazon's SQS and SNS, may not support durable subscriptions.

Last updated: March 16, 2023