How does autoscaling work with Lambda functions triggered by SQS?

How to, AWS By Aug 22, 2023 No Comments

Lambda autoscaling with SQS triggers works as follows:

When an SQS queue is configured as a trigger for a Lambda function, Lambda will start by polling the queue using 5 concurrent connections. As messages become available in the queue, Lambda will spin up Lambda functions to process the messages.

Initially, Lambda will start processing messages in batches of up to 10 messages at a time, with up to 5 concurrent invocations of your Lambda function. As more messages become available, Lambda will continue to scale up the number of function instances as follows:

The scaling behavior depends on several factors:

  • The batch size you configure: A larger batch size means fewer invocations but each invocation processes more messages.
  • The batch window: If you set a batch window, Lambda will continue polling for messages until that window expires, even if the batch size is reached earlier.
  • The number of messages in the queue: As the queue grows, Lambda will scale up to match the queue size.
  • The maximum concurrency setting: You can configure a maximum concurrency to limit how many Lambda instances the SQS trigger can invoke simultaneously.

When a message processing fails, Lambda will automatically retry the message while implementing a backoff strategy. It will gradually reduce the amount of concurrency allocated to the SQS event source mapping.

In summary, Lambda autoscaling with SQS triggers allows Lambda to dynamically scale to match the throughput required to process all messages in the SQS queue in a reasonable time. The batch size, batch window, maximum concurrency and queue size all influence how quickly and extensively Lambda scales its function instances.



I'm Abhay Singh, an Architect with 9 Years of It experience. AWS Certified Solutions Architect.

No Comments

Leave a comment

Your email address will not be published. Required fields are marked *