How does using SQS help optimize costs compared to invoking Lambda directly from SNS?

Optimizing costs of invoking Lambda from SQS vs SNS

There are a few ways that using SQS as a trigger for Lambda functions can optimize costs compared to invoking Lambda directly from SNS:

1. Batching messages – SQS allows you to batch messages and invoke Lambda with multiple messages at once. This reduces the number of Lambda invocations and thus the costs. You can configure a batch size of up to 10,000 messages for standard queues and 10 messages for FIFO queues.

--batch-size 100

2. Longer Lambda durations – Since SQS buffers messages, Lambda functions can take longer to process batches of messages. This reduces the total number of invocations and costs.
3. Controlled scaling – SQS allows you to control the rate at which Lambda scales up to process messages. This helps avoid unnecessary scaling events that incur costs.
4. Data transfer costs – Invoking Lambda directly from SNS can incur data transfer costs as each message is passed to Lambda. Using SQS as an intermediary avoids these costs.
5. Retry logic – SQS provides features for retrying and reprocessing failed messages. This helps ensure messages are eventually processed successfully, avoiding wasteful invocations.
6. Idle time – When no messages are in the queue, Lambda functions are idle and not incurring costs. With SNS, Lambda functions are always “waiting” to be invoked.

In summary, using SQS as a trigger for Lambda allows you to batch messages, control scaling, implement retry logic, and avoid data transfer costs. All of these factors can help significantly optimize the costs of running Lambda functions compared to invoking Lambda directly from SNS.

Sources

1. https://stackoverflow.com/questions/42656485/sns-to-lambda-vs-sns-to-sqs-to-lambda
2. https://stackoverflow.com/questions/72781519/what-are-the-benefits-of-using-sns-to-sqs-to-lambda-compared-to-having-sns-to-sq
3. https://docs.aws.amazon.com/lambda/latest/operatorguide/cost-optimize.html