0

I'm running Node 14 on an AWS ECS machine with has 5120 CPU and 8192 Memory. At times when running on heavy load (either compressing large number of images using imagemin or copying files from S3 to the local machine using S3 sync) I get the ENOMEM error.

This causes the ECS instance to crash.

The S3 sync command is executed via child_process spawn, while imagemin package also spawns a child process in order to compress images. Both errors are displayed below

(node:23) UnhandledPromiseRejectionWarning: Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:403:11)
    at spawn (child_process.js:553:9)
    at new SpawnTimeout (/app/src/utils/SpawnTimeout.ts:33:25)
    at s3Cp (/app/src/utils/s3.utils.ts:62:24)
    at copyFile (/app/src/utils/s3.utils.ts:52:10)
    at executeImageCompression (/app/src/processors/Processor.ts:63:9)

enter image description here

I tried increasing the ECS resources and adding custom -max-old-space-size but it did not help.

I saw this post - Node.js catch ENOMEM error thrown after spawn

But I cannot configure --memory-swap param in the docker as the deployment is controlled by someone else.

Please advise on how I can resolve this?

David Faizulaev
  • 4,651
  • 21
  • 74
  • 124
  • Hi, can you show us what command / option and its value you used when you tried to increse the ECS resources ? – MaieonBrix Dec 30 '21 at 08:51
  • I created a new task definition and updated the values there. – David Faizulaev Dec 30 '21 at 08:53
  • It's because Node.js(V8) failed to spawn new process due to the system OS lack of memory. Please note that each child process is a standalone process and consumes quite a lot of memory. – iKoala Dec 30 '21 at 09:24
  • @iKoala I tried to limit the number of concurrent child processes to 25 instead of 100, but still got the same error. – David Faizulaev Dec 30 '21 at 12:03

1 Answers1

1

Memory is allocated to every child_process.spawn() that you call. From what I see, imagemin could be calling child_process.spawn() more times than the computer can handle, leading to nodejs allocating more memory than the computer has to the newly spawned child_processes, leading to the ENOMEM error. Either that, or the processes may be taking up too much memory. This is just my speculation, as I can't see the code.

You could try decreasing or putting a limit on the number of new child_processes being spawned.

Again, this solution is purely speculative as I have no access to your code, so it would be helpful if you could add it to your question.

This link may further help you: How can I limit the number of child proccesses in imagemin?

user9002871
  • 15
  • 3
  • I tried to limit the number of concurrent child processes to 25 instead of 100, but still got the same error. – David Faizulaev Dec 30 '21 at 11:57
  • Best practice is limiting the number of child processes to the number of CPU cores that your machine has. Adding more child processes beyond that will not add much performance difference. If you are already doing that, you could check how much memory each child process is using with this [library](https://stackoverflow.com/questions/10730429/how-to-get-a-child-process-memory-usage-in-node-js) at its peak, then divide the total memory with the memory needed for each child. Allocate some "buffer" memory as well to ensure the child processes do not take up all the memory. Hope this helps! – user9002871 Dec 31 '21 at 06:30