Dealing with AWS SQS duplicate messages
Not so long time ago, my ex-colleague asked, why he receives duplicate messages with AWS SQS?
The general architecture was pretty common: there`s a process that pushes the message to the queue that some task should be done, and several processes that can pick up messages and process them. The idea behind that was that messages should be processed only once. But, unfortunately, the message was duplicated and processed several times.
So, what was the reason for it?
Although it's a common issue, there might be several reasons for it.
First of all, SQS by design guarantees at least one delivery. It means that the same message can be delivered several times. For exactly-once delivery, SQS has FIFO Queues with Exactly-Once Processing. For some reason, it was not possible to use it in the current setup.
But how do we still can decrease the number of duplicates using Standart Queues?
For that, we should take a look at the Visibility Timeout property of the queue. During this timeout, SQS prevents other consumers from reading the message. When timeout is reached, message put back in queue and another consumer can read it in start processing.
The maximum value for visibility timeout is 12 hours, and the optimal value is the time during which your consumer processing the message. Of course, you should add some gap for it in case if that process takes longer.
Few tips that might help you to achieve exactly one delivery with SQS Standart Queues:
- Do not forget to delete the message once it processed
- To check if the message is already processed by another worker, you can create an additional check on the unique message id. If the message-id is already in storage\cache, do not pick it up. (of course, it`s a bit of overhead and a better option is to consider using FIFO with exactly one delivery mentioned previously)
- It is possible to set the visibility timeout on a specific message, not only on the entire queue. It can be done using ChangeMessageVisibility API operation
In general, my suggestion would be to use tools that are designed for particular use cases. In the situation mentioned above, there should be a strong reason against using FIFO Queues with Exactly-Once Processing and a good understanding of additional complexity and overhead for a customized solution.
