Parallel lambdas run slowly

Question

I'm trying to run a process launching 1000 lambdas in parallels.

I made a test with 10 lambdas that succeeded in ~4 seconds, however scale up to 1000 lambdas increase the running time to ~23 seconds.

const lambda = new AWS.Lambda();
await Promise.all(
    Array(1000)
      .fill(0)
      .map((_, i) => {
        if (i === 0) console.log("start lambda", i, new Date().getSeconds());
        if (i === 999) console.log("start lambda", i, new Date().getSeconds());
        return lambda
          .invoke({
            FunctionName: "myFunction",
            Payload: JSON.stringify(payload),
          })
          .promise()
          .then((x) => {
            if (i === 0) console.log("end lambda", i, new Date().getSeconds());
            if (i === 999) console.log("end lambda", i, new Date().getSeconds());
            return x;
          });
      })
);

output

start lambda 0 11
start lambda 999 11
end lambda 0 15
end lambda 999 39

I'm quite surprised about that observation because Lambda is a service designed to be highly scalable. I'd like to understand why is it so slower and what could I do to solve this problem.

ok, the reason I asked is because depending on the region, you'd have different quotas. I think this is useful: https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html Do you have other lambda running in this account? — brushtakopo, Nov 08 '22 at 13:37
I suspect this is a limitation of the program that you're using to invoke the Lambdas, rather than Lambda iteself. What happens if you replace the invoke request with an HTTP GET to some other (fast-responding) endpoint? — kdgregory, Nov 08 '22 at 13:40
@AnthonyB. We raised the ConcurrentExecution Quotas. Also, it seems those lambdas should throw 429 error when they are throttled, which seems not happening — Antonin Riche, Nov 08 '22 at 13:55
@AntoninRiche, have you looked at the lambda monitor in aws console? That could give you great information. Also, enable x-ray might help. — brushtakopo, Nov 08 '22 at 14:01
Note that in the absence of `InvocationType="Event"` you're [invoking](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#invoke-property) the Lambda functions synchronously. They're going to run in series, not in parallel. Was that your intention? — jarmod, Nov 08 '22 at 14:55
There is something odd about your code, specifically `Array(1000).map(...)`. That code will not map over anything because, while you initialized the array to 1000 empty items, the map callback is only invoked for indexes of the array that have assigned values, and there are none in Array(1000). Is the code you've posted the actual code you are running? Or has it been transpiled in some way? Which version of Node.js are you using? Related post [here](https://stackoverflow.com/questions/5501581/javascript-new-arrayn-and-array-prototype-map-weirdness). — jarmod, Nov 08 '22 at 17:32
@jarmod my bad sorry, I simplified the code to post and I removed the .fill thinking it was extra. but indeed you're right. — Antonin Riche, Nov 08 '22 at 22:20

brushtakopo · Accepted Answer · 2022-11-08T15:25:29.700

I think you are being limited by the SDK. Nothing to do with Lambda.

By default, the SDK allow 50 requests to run in parallel.

In your code, you have to change the maxSockets to something that works for you.

Something like this:

var AWS = require('aws-sdk');
var https = require('https');
var agent = new https.Agent({
   maxSockets: 1000
});

const lambda = new AWS.Lambda({
   apiVersion: '2012-08-10'
   httpOptions:{
      agent: agent
   }
});

But in my opinion, it is not a good way to test your lambda function execution. Using Lambda monitor, Lambda Insights and AWS X-Ray is a better way to look at it.

Alternatively, I will encourage you to deploy this solution to see if your lambda can be fined tuned: https://github.com/alexcasalboni/aws-lambda-power-tuning

It is not because you chose the 128MB memory that your lambda will be cheaper, you can be very surprised by the results with higher memory.

Parallel lambdas run slowly

1 Answers1