Rails concurrent requests

Question

So I have a rails app that parses a long json file - during which it sends off requests to an external API, which seem to take forever. My question is, is it as simple as to stick each request into a separate thread in order to speed it up? I don't see why I can't have say 3 or 4 threads running concurrently sending off requests ... the only issue I see is currently the method displays an errors page if there are any errors present at the end of parsing - and this possibly couldn't happen as the other threads might still be running? .. eventually I want to make this method into a rake task and run it using a cron job that renders a html page so that should be better correct?

def load_movies
 @errors = []
 movie_list = get_data

 movie_list.first(50).each do |movie_data|
 ------------ Thread.new do -----------
  movie = Movie.new
  movie.construct_movie movie_data

  if !movie.valid?
    @errors << movie
  else
   movie.save
  end
  ----------- end thread ------------
  unless @errors.count > 0
  redirect_to root_path
end
end

is it as simple as to do what is above and it will run off open multiple threads to send the requests (which can be assumed to be in construct_movie

This sounds like a perfact example for a background job. Did you consider using [DelayedJobs](https://github.com/collectiveidea/delayed_job) or [Sidekiq](http://sidekiq.org/)? — spickermann, Feb 15 '15 at 22:55
I did look at sidekiq - my issue was the whole having it redirect to an errors page at the end, I didn't think I could do that. — Carl Wright, Feb 15 '15 at 23:12
You could collect errors in the database and display them to the user once all jobs are done. — spickermann, Feb 15 '15 at 23:17
How do I detect when all jobs are done and then do something? — Carl Wright, Feb 16 '15 at 00:23
Have each job report back its status to the model in the database. If the number of jobs reported back equals the number of jobs created, obviously all jobs are done... — spickermann, Feb 16 '15 at 01:09

score 3 · Accepted Answer · answered Feb 15 '15 at 23:44

You can do this, yes. I have done it, yes.

You can run into some gotchas with multi-threaded use of ActiveRecord, if you're using ActiveRecord. For doing multi-threaded stuff in ruby, I recommend the concurrent-ruby gem, which provides thread pools and higher level abstractions like Futures. https://github.com/ruby-concurrency/concurrent-ruby

You can do this, although, yeah, it can get tricky -- if a background job queue works, that might be a less reinventing-the-wheel solution for you. Whether a background job queue or threads, you're going to have to figure out how to display 'results' -- although with the straight threads approach, I guess your request can wait on all the threads and their results, before displaying results (yeah, I've done this too). That can cause it's own problems with having slow-running requests (you'll probably want to deploy with puma or passenger enterprise to avoid blocking your entire app process on them).

From my own experience doing similar things, I don't have an easy answer, you'll run into some challenges no matter what route you take, I think. If you can figure out a way to make a background job queue work, I think that will generally give you less gotchas than directly creating threads in the app process.

Rails concurrent requests

1 Answers1

Linked