Wait for SuckerPunch::Job task completion

Question

I try the Sucker Punch gem to process tasks in parallel. But I found no documentation how to wait for its termination.

require 'sucker_punch'

class SuckerJob
  include SuckerPunch::Job
  workers 4

  def perform(event)
    sleep(rand(5))
    puts "[#{Thread.current.object_id}] End processing event   #{event}."
  end
end

10.times { |i| SuckerJob.perform_async(i) }
puts "Shutting down ..."
SuckerPunch::Queue.shutdown_all
puts "Shutdown finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"
# [Ugly] call internal method
SuckerPunch::Queue::QUEUES.fetch_or_store(SuckerJob.name).wait_for_termination(10)
puts "Wait finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"

It seems, that SuckerPunch::Queue.shutdown_all() returns before all tasks are completed.

Shutting down ...
[17487240] End processing event   1.
[17488760] End processing event   0.
[17487240] End processing event   4.
[17488760] End processing event   5.
[17486120] End processing event   2.
[17484940] End processing event   3.
[17487240] End processing event   6.
Shutdown finished, status: {"workers"=>{"total"=>3, "busy"=>3, "idle"=>0}, "jobs"=>{"processed"=>7, "failed"=>0, "enqueued"=>0}}
[17484940] End processing event   9.
[17488760] End processing event   7.
[17486120] End processing event   8.
Wait finished, status: {"workers"=>{"total"=>0, "busy"=>0, "idle"=>0}, "jobs"=>{"processed"=>10, "failed"=>0, "enqueued"=>0}}

How can I wait until all tasks are completed?

you can try `SuckerPunch::Counter::Processed.new(@queue).value` — Rajarshi Das, Mar 14 '17 at 12:58
or check stats you can do `all_stats = SuckerPunch::Queue.stats` then `stats = all_stats[MyJob.to_s]` then stats you get now you can see `stats["jobs"]["processed"]` now there is `total`, `processed`,`busy`.... — Rajarshi Das, Mar 14 '17 at 13:03

score 3 · Answer 1 · answered Mar 14 '17 at 13:07

You can check status

or check stats you can do

all_stats = SuckerPunch::Queue.stats

then

stats = all_stats[SuckerJob.to_s]

then stats you get now you can see

stats["jobs"]["processed"]

you can check like below

  stats["jobs"]["processed"] > 0
  stats["jobs"]["failed"] == 0
   stats["jobs"]["enqueued"] == 0

score 0 · Answer 2 · answered Nov 09 '21 at 23:25

I use this:

def wait_for_jobs(job_name:, count:, max_seconds: 100)
  Rails.logger.info "Waiting up to #{max_seconds} seconds for #{count} jobs to run"
  wait_time = 0
  while wait_time < max_seconds
    stats = SuckerPunch::Queue.stats[job_name]
    processed = stats['jobs']['processed']
    break unless processed < count

    sleep(1)
    wait_time += 1
  end
  raise StandardError, "Timeout while waiting for #{count} jobs of #{job_name} to have run!" unless wait_time < max_seconds

  Rails.logger.info "#{count} jobs took #{wait_time} seconds to run"
end

Wait for SuckerPunch::Job task completion

2 Answers2