What is the benefit of using SQS as an intermediary between SNS and Lambda?

Using an SQS queue as an intermediary between SNS and Lambda has several benefits:

1. Batch processing – Lambda can read messages from the SQS queue in batches and invoke a single function to process multiple messages. This reduces the number of Lambda invocations and can optimize costs.
2. Rate limiting – The SQS queue acts as a buffer and allows you to control how fast messages are processed by Lambda. This can prevent overloading Lambda if the SNS publish rate is high.
3. Retries and reprocessing – If Lambda fails to process a message, that message remains in the SQS queue and can be retried. You can configure the number of retries. With SNS, failed messages are lost after a few retries.
4. Persistence – Messages in SQS queues can persist for up to 14 days. This allows Lambda to process messages even if it is temporarily unavailable.
5. Scaling – Lambda will automatically scale to match the number of messages in the SQS queue. This provides a self-scaling architecture.
6. Latency – SQS introduces some latency as Lambda has to periodically poll the queue for new messages. However, this gives you more control over message processing.
7. Ordering – SQS Standard queues provide FIFO message ordering which SNS does not provide. This ensures messages are processed in the order they were received.

In summary, using an SQS queue between SNS and Lambda gives you more control over message batching, rate limiting, retries, scaling and ordering. The trade-off is some additional latency and complexity. But for most use cases, the benefits outweigh the downsides.

Sources

1. https://stackoverflow.com/questions/42656485/sns-to-lambda-vs-sns-to-sqs-to-lambda
2. https://beabetterdev.com/2021/09/10/aws-sqs-vs-sns-vs-eventbridge/
3. https://plainenglish.io/blog/aws-sns-to-lambda-vs-sns-to-sqs-to-lambda-788d4cc96f34