1

I have a queue example from here (Python+Tornado framework): https://www.tornadoweb.org/en/stable/queues.html

Now it's a sequential queue. How to make it parallel?

Since I don't fully understand tornado.queues now, it's not clear for me how should the code be changed to implement a parallel queue. 

from tornado import gen
from tornado.ioloop import IOLoop
from tornado.queues import Queue

q = Queue(maxsize=2)

async def consumer():
    async for item in q:
        try:
            print('Doing work on %s' % item)
            await gen.sleep(0.01)
        finally:
            q.task_done()

async def producer():
    for item in range(5):
        await q.put(item)
        print('Put %s' % item)

async def main():
    # Start consumer without waiting (since it never finishes).
    IOLoop.current().spawn_callback(consumer)
    await producer()     # Wait for producer to put all tasks.
    await q.join()       # Wait for consumer to finish all tasks.
    print('Done')

IOLoop.current().run_sync(main)

I expect all the work to start simultaneously and then to finish simultaneously instead of doing tasks one by one.

Thanks a lot!

ZenBerry
  • 217
  • 2
  • 12
  • This is the expected behaviour because Tornado is single threaded. Hence, the IOLoop can only run one thing at a time. For more, see [differences between concurrency vs parallelism](https://stackoverflow.com/questions/1897993/what-is-the-difference-between-concurrent-programming-and-parallel-programming). – xyres Jun 05 '19 at 06:48

1 Answers1

1

All you need to do is spawn multiple consumer tasks:

for i in range(num_consumers):
    IOLoop.current().spawn_callback(consumer)

Then each consumer will be able to read from the queue and await things in parallel. (Note that because Tornado is single-threaded, anything that does not use await will block everything)

Ben Darnell
  • 21,844
  • 3
  • 29
  • 50
  • Shouldn't he use `add_callback` instead of `spawn_callback` since the previous is just an allias ? From your own docs https://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.IOLoop.spawn_callback ;) (ps: Thanks for everything you've done with Tornado, I love it) – NicoAdrian Jul 14 '19 at 10:45
  • `add_callback` and `spawn_callback` are equivalent now, but they were different in older versions of tornado. I used `spawn_callback` here because the question did. – Ben Darnell Jul 16 '19 at 03:13