How to execute a job on a server on Lambda without waiting for the response?

Question

I am trying to spawn a mapreduce job using the mrjob library from AWS Lambda. The job takes longer than the 5 minute Lambda time limit, so I want to execute a remote job. Using the paramiko package, I ssh'd onto the server and ran a nohup command to spawn a background job, but this still waits until the end of job. Is there anyway to do this with Lambda?

score 0 · Answer 1 · answered May 25 '17 at 16:07

0

In python you can launch a shell command without waiting for its end. Here you'll find more information

answered May 25 '17 at 16:07

Sébastien Coste

309
2
11

score 0 · Accepted Answer · answered May 27 '17 at 06:34

0

Instead of using SSH I've solved this before by pushing a message onto a SQS queue that a process on the server monitors. This is much simpler to implement and avoids needing to keep credentials in the lambda function or place it in a VPC.

answered May 27 '17 at 06:34

Robo

992
4
10

This exactly what I ended up doing and then had the ability to use roles. – user2820906 Jun 06 '17 at 20:23

How to execute a job on a server on Lambda without waiting for the response?

2 Answers2