133

I am currently using Resque for my background process but recently I heard a lot of huff-buff about sidekiq. Could anybody compare/differentiate?

In particular I would like to know is there a way to monitor programmatically whether a job is completed in sidekiq

Sergio Tulentsev
  • 226,338
  • 43
  • 373
  • 367
Bhushan Lodha
  • 6,824
  • 7
  • 62
  • 100
  • 74
    Am so psyched to see that this question has not been closed as being "not appropriate for SO". – Dogweather Nov 09 '12 at 05:23
  • 54
    Can we stop closing all the good question please people. I know it's quite broad, it's also quite important and quite interesting. – superluminary Sep 18 '14 at 13:11
  • 21
    One day they will invent a site where programmers can ask questions, and that site will allow subjective responses. Questions that can be ignored, or even voted on based on relevance. – baash05 Sep 25 '14 at 01:34
  • 4
    @baash05 and it will be called Quora! – nakhli Nov 06 '14 at 01:09
  • 6
    It will be called Yahoo Answers, and the quality will plummet through the floor. – halfer Mar 20 '15 at 21:40
  • Ryan has created a webcast for Sidekiq this week which also include some comparison with Resque. You probably want to check it out: Url: http://railscasts.com/episodes/366-sidekiq – Firoz Ansari Jul 20 '12 at 14:18

2 Answers2

135

Resque:

Pros:

Cons

  • runs a process per worker (uses more memory);
  • does not retry jobs (out of the box, anyway).

Sidekiq:

Pros

  • runs thread per worker (uses much less memory);
  • less forking (works faster);
  • more options out of the box.

Cons

  • [huge] requires thread-safety of your code and all dependencies. If you run thread-unsafe code with threads, you're asking for trouble;
  • works on some rubies better than others (jruby is recommended, efficiency on MRI is decreased due to GVL (global VM lock)).
Sergio Tulentsev
  • 226,338
  • 43
  • 373
  • 367
  • 21
    How do you know if you're running "thread unsafe code"? – Dogweather Nov 09 '12 at 05:24
  • 4
    For example, if you're storing state in global variables, then you're in trouble :) As for the gems, here are some [in the wiki](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting). – Sergio Tulentsev Nov 09 '12 at 06:35
  • If you're using MRI (official ruby) everything is always thread-safe, because of the GVL. So you are safe if you are using MRI (but performance may be worse depending on what you're doing). – mrbrdo Apr 08 '13 at 19:58
  • 25
    @mrbrdo You can't be more wrong. GVL has nothing to do with thread-safety of your ruby code. GVL is about thread-safety of ruby interpreter code (MRI C code). – radarek Jun 14 '13 at 09:02
  • 2
    I have to contradict the 'pro' on Resque that says 'you can use any ruby'. The README for Resque today actually says 'We would love to support non-MRI Rubies, but they may have bugs.' https://github.com/resque/resque/blob/master/README.md#requirements – JellicleCat Dec 11 '13 at 21:00
  • @JellicleCat can you do the edit? Strike-through that point and add comment about resque now being MRI-only (with link) – Sergio Tulentsev Dec 18 '13 at 01:48
  • 2
    I would particularly avoid Sidekiq if you're planning to run JavaScript inside your jobs using therubyracer. Not a fault of Sidekiq per se but the multi-threading causes issues for therubyracer. See https://github.com/cowboyd/therubyracer/issues/206 – Jeremy Burton Aug 15 '14 at 19:31
  • Where's the reference link for "jruby and rubinius are recommended" for SideKiq? – Ryan Buckley Jan 21 '16 at 18:04
  • @RyanBuckley: hm, weird. Rubinius used to be recommended. But now can't find a mention of it anymore in github repo. – Sergio Tulentsev Jun 10 '16 at 05:03
  • being able to run multiple processes in Resque, makes it faster in multicore systems. – vasilakisfil Aug 08 '18 at 09:29
  • @vasilakisfil: does it offset the cost of forking? :shrug: This needs to be properly measured. Besides, you can simply run several sidekiq processes. – Sergio Tulentsev Aug 08 '18 at 09:44
  • @SergioTulentsev Ofcourse the longer the jobs the better. But usually a job split in 2+ processes that run parallely will always be faster than 1. – vasilakisfil Aug 09 '18 at 10:07
10

From the question:

In particular I would like to know is there a way to monitor programmatically whether a job is completed in sidekiq

Here's a solution for that:

  1. Sidekiq::Status gem
  2. Batch API (Sidekiq Pro) - usage
Gurpartap Singh
  • 2,744
  • 1
  • 26
  • 30