How to get passed parameters to return values in multiprocessing Pool?

Question

For example:

from multiprocessing import Pool
from functools import partial

numbers = []
for i in range(100):
    numbers.append(i)

def add_one(number, new_numbers):
    new = number + 1
    new_numbers.append(new)

new_numbers = []
process_pool = Pool(2)
add_one_helper = partial(add_one, new_numbers=new_numbers)

process_pool.map(add_one_helper, numbers)

print(new_numbers)

I suppose the new_numbers will print: 1, 2, 3, 4..., but it is empty in print(). How to get new_numbers to be populated after the call?

If i add process_pool.join(), it will error: 'ValueError: Pool is still running' — marlon, Oct 04 '21 at 08:22
Could be caused by [python - Why multiprocessing.Pool cannot change global variable? - Stack Overflow](https://stackoverflow.com/questions/53878553/why-multiprocessing-pool-cannot-change-global-variable) — user202729, Oct 04 '21 at 08:22
Either way this is completely the wrong way to use `Pool.map`. Why can't you do it the normal way? — user202729, Oct 04 '21 at 08:34
This answer doesn't work: https://stackoverflow.com/questions/11025005/python-sharing-a-dictionary-between-parallel-processes/11025090#11025090, TypeError: f() missing 1 required positional argument: 'x' — marlon, Oct 04 '21 at 08:34
@user202729 what's the normal way? please write an answer. The linked answer gave an error. — marlon, Oct 04 '21 at 08:35

score 0 · Accepted Answer · answered Oct 04 '21 at 12:42

First an observation. You have:

numbers = []
for i in range(100):
    numbers.append(i)

This is using a loop to create a list with values 0, 1, 2 ... 99. But you could have just as easily and more efficiently specified:

numbers = list(range(100))

But in fact the iterable argument being passed to the Pool.map method doesn't need to be a list (it will, however, be automatically converted to a list if it does not support the __len__ method). So instead of creating the numbers list, you could have simply passed instead range(100) as the iterable argument to the map call. Now on to the question at hand:

The Pool.map method returns a list of all the return values from calling the worker function (add_one in this case). So you just need to have add_one return the value it wants appended to the new list:

from multiprocessing import Pool

def add_one(number):
    return number + 1

process_pool = Pool(2)
new_numbers = process_pool.map(add_one, range(100))
print(new_numbers)

Prints:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]

How to get passed parameters to return values in multiprocessing Pool?

1 Answers1