1

I have read Python multiprocessing.Pool: when to use apply, apply_async or map? it was useful, but still had my own questions. In the following code, I want result_list.append(result) in a parallel way, I want the 4 processors to append the result parallelly and convert 4 lists to 1 list.

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool(4)
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()
Community
  • 1
  • 1
shao
  • 11
  • 1
  • 3
  • Is it easier to produce 4 lists with 4 processors and afterwards merge 4 lists to 1 list? – Wenlong Liu Feb 28 '17 at 03:49
  • That's exactly what I want,how to implement it? – shao Feb 28 '17 at 05:52
  • 1
    I'm not sure I understand what you're asking. It sounds like you want `pool.map`. There's no way for the child processes to do the appending to the result list, since the list lives in the main process's memory (which is not shared with the other processes). There are some synchronized types that might work with some extra effort (e.g. `multiprocessing.Array`), but I suspect using them safely would require quite a bit of overhead. Using `pool.map` is much easier and is probably faster than a similar system you put together yourself. – Blckknght Feb 28 '17 at 06:28

1 Answers1

2

Multiprocessing pool will be your choice.

Following are some sample code, hope it helps. You can also check another my answer to see more details. How can I make my python code run faster

    from multiprocessing import Pool
    import time

    def foo_pool(x):
        return x*x

    def main():
         pool = Pool(4)
         sampleData = [x for x in range(9)]
         results = pool.map(foo_pool, sampleData)
         pool.close()
         pool.join()
         print(results)

    if __name__ == '__main__':
         main()
Community
  • 1
  • 1
Wenlong Liu
  • 444
  • 2
  • 13