1

I am running a queue process using Amazons SQS and a separate machine that processes the jobs (a worker). I am also using the supervisor to make sure the queue:listen is always running on the worker machine, yet when I define numprocs=8 (like the example on laravel's website) jobs are being executed more than once hence emails are being sent couple of times etc'.

Any Idea how can I make sure a job is executed only once even if I am running multiple worker processes and machines?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Broshi
  • 3,334
  • 5
  • 37
  • 52

1 Answers1

0

With SQS, once a worker takes an item - it is the worker's responsibility to immediately delete the item from SQS so that other workers will not get the item as well.

This part of the documentation explains what amount of time you are allowed for, to delete the item - http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html

Since you did not show the code of your e-mail sending Laravel worker, I can only guess that you are not deleting the SQS item immediately - but instead maybe deleting it only after the e-mail was sent. Should the e-mail sending part take more than SQS Visibility Timeout (described in the link document), it would make sense the same task is picked up by other workers.

Evgeny
  • 6,533
  • 5
  • 58
  • 64
  • Also, request counter will increase when next worker request it. It is a good indicator used for cleanup or rerun some task. – mootmoot Jun 10 '16 at 13:41