multiprocessing apply_async strange behaviour with multiprocessing queue

Question

import time
import multiprocessing

def multi_thread(files):
    q = multiprocessing.Queue()

    for f in files:
        q.put(f)

    p = multiprocessing.Pool(5)

    for i in range(5):
        p.apply_async(worker_test, args=(q,))

    p.close()
    p.join()


def worker_test(i):
    print 'hello'
    print i


def main():
    files = ['a', 'b', 'c', 'd']

    multi_thread(files[0:4])
    print 'Multi-thread time: {0} seconds'.format(time.time() - t0)

if __name__ == '__main__':
    main()

My code doesn't even enter the work_test() function to print hello if I pass in q. But if I change q to i, the code runs fine. Somehow it doesn't like the multiprocessing.Queue() object - any ideas what's going on?

See [this question](http://stackoverflow.com/questions/3827065) — Janne Karila, Feb 05 '15 at 09:07

score 1 · Accepted Answer · answered Feb 05 '15 at 07:20

1

You need to create a queue instance that can be shared among different processes. You can do this by using multiprocessing.Manager object.

The following code seems to work:

import time
import multiprocessing

def multi_thread(files):
    m = multiprocessing.Manager()
    q = m.Queue()

    for f in files:
        q.put(f)

    p = multiprocessing.Pool(5)
    for i in range(5):
        p.apply_async(worker_test, args=(i, q))

    p.close()
    p.join()

def worker_test(i, q):
    print 'hello'
    print i

def main():
    files = ['a', 'b', 'c', 'd']

    multi_thread(files[0:4])

if __name__ == '__main__':
    main()

answered Feb 05 '15 at 07:20

Ozgur Vatansever

49,246
17
84
119

Thanks for the reply, that works. However, my `Queue` works fine though if I start each `Process` manually instead of using `Pool`. Why is it that with only with `Pool` I need to use `multiprocessing.Manager`? – mchangun Feb 05 '15 at 07:26
1

currently I don't know the difference between a process created outside of a pool and a process created within a pool. I've always used manager regardless of what kind of parallel calculation I am doing, because you are very likely to share some values among processes so you'll have to have a manager eventually. – Ozgur Vatansever Feb 06 '15 at 01:45

multiprocessing apply_async strange behaviour with multiprocessing queue

1 Answers1