Python mutiprocessing pool with partial arguments

Question

So I'm trying to follow the example here: Python multiprocessing pool.map for multiple arguments

What if I have three arguments and I need the third fixed to a value, e.g. some thing like this but it doesn't work. Raise TypeError saying partial_merge() takes exactly 2 arguments but only 1 given.

import multiprocessing
from functools import partial


def merge_names(a, b, c):
    return '{} & {} & {}'.format(a, b, c)


if __name__ == '__main__':
    names = [('Brown', 'Wilson'), ('Bartlett', 'Rivera'), ('Molloy', 'Opie')]
    pool = multiprocessing.Pool(processes=3)

    results = pool.map(partial(merge_names, c='Hello'), names)
    pool.close()
    pool.join()
    print(results)

Answer 1

pool.map() works like the 'normal' map(): it takes a function with one missing argument and a list and applies the function to each element of the list. The easiest way to solve your problem would be to modify your function:

def merge_names(tup, c):
    a, b = tup
    return '{} & {} & {}'.format(a, b, c)

---

Depending on your goal/setting, there are other possibilities (increasing in complexity):

1. closures. Example:

   def func(x):
       return x + y
   
   def main():
       y = 5  # the point is to declare this BEFORE Pool() is initiated
       p = mp.Pool(3)
       data = range(10)
       results = p.map(func, data)
   

2. shared values: see the documentation on sharing state

3. queues and pipes: see the documentation on queues and pipes

4. not using pool but mp.Process() which allows you much finer control of arguments and more.