How to do FIFO messaging with Amazon SQS

If you have used Amazon Web Services, you probably know Simple Queue Service(SQS) - it is a reliable, highly scalable hosted queue for storing messages. One of the main drawbacks of SQS is that it does not guarantee first-in, first-out (FIFO) access to messages and that's clearly stated in the Amazon documentation:

"Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it."

Here is a quick example how Camel Resequencer pattern can help you overcome this drawback in 2-3 lines of code. To setup our example scenario, let's first create a route that will populate our queue with 100 messages each containing the message number: Then create a consumer route, that will read from the queue and log each message:
To prove that Amazon doesn't guarantee FIFO ordering we will write a test: Don't be misled by the short size of the test, it starts both routes and verifies that all 100 messages are received in the right order.

The above test fails in most of the runs and proves that SQS doesn't support FIFO order(when the messages are sent too quickly). The idea in this example is that our producer will index the messages or provide some kind of sequencing information, so that the message consumer can interpret it and order the messages. To do that we simply send a number as message body, but in a real world application that can be a field in a JSON or XML message. Then to make the test pass and ensure that the messages are received in the same order as they were sent, all we have to do is to add the Resequencer pattern in our consumer route: The streaming based Resequencer will let the messages go without any delay as long as they are in the right order. If the messages are not in the right sequence, it can hold up to 100 messages for 1 second while waiting for the missing message. Depending on your message load, you should adjust these numbers to hold enough messages while waiting for the missing one, but not hold for too long and reduce the throughput. Another option would be to try the non-streaming batch based Resequencer, which always collects a number of messages before sorting and releasing them.
If you are new to Apache Camel and Enterprise Integration Patterns (like the Resequencer), have a look at my recently published "Instant Apache Camel Message Routing" book where similar patterns and how to use them in Camel are explained in a short and focused manner.
Or if you want to deep dive into the integration world, I recommend you start from "Enterprise Integration Patterns" and "Camel in Action" books.

Posted in: AWS,Camel,EIP