Multiprocessing pool map doesn't accept several arguments as list of lists

Question

How can I pass several argument to a pool.map correctly. Currently I have an error saying :

File "C:/Users/maxime/Desktop/execom/di.py", line 42, in <module>
    A = pool.map(energy2, args)
  File "C:\Python36\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Python36\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
TypeError: energy2() missing 2 required positional arguments: 'window' and 'i'

zip doesn't work neither. I also try something with partial but if all arguments need to be defferent in each process, it is then pointless.

import numpy as np
from numba import njit
import multiprocessing
from functools import partial
from itertools import repeat
import itertools


def energy2(signal, window,i ):
    L2 = int(len(window ) /2)
    Lw = len(window)
    taille = len(signal)
    channel_buffer = np.zeros(len(signal))
    filtered_signalI = np.hstack((np.zeros(L2) ,signal ,np.zeros(len(window))))
    for k in range(0 ,taille):
        buffer = (filtered_signalI[k : k + Lw ] * window)
        channel_buffer[k] = np.sqrt(np.sum(buffer * buffer))
    return channel_buffer / Lw



if __name__=="__main__":
    multiprocessing.freeze_support()
    # serial
    window = np.random.rand(32)
    N = 10
    Signals = np.zeros((N,1000))
    for i in range(N):
        Signals[i,:] = np.random.rand(1000)
    Res = np.zeros(Signals.shape)
    for i in range(N):
        Res[i, :] = energy2(Signals[i, :], window,i)
    print(Res)
    # parallel
    cpu_nb = 11  # multiprocessin
    Res2 = np.zeros(Signals.shape)
    idx = 0
    pool = multiprocessing.Pool(cpu_nb)
    args = []
    for h in range(N):
        args.append([window,Signals[h, :],h])
    A = pool.map(energy2, args)
    for imap in range(len(A)):
        Res2[imap, :] = A[imap]


    print(Res - Res2) #find same results

score 3 · Accepted Answer · answered Mar 09 '20 at 15:02

I think the answer to this lies in starmap. See this for a previous solution. I'm not sure on the format of your inputs, but my guess is that you will need to zip them if you do indeed have a list of lists.

if __name__=="__main__":
    multiprocessing.freeze_support()
    # serial
    window = np.random.rand(32)
    N = 10
    Signals = np.zeros((N,1000))
    for i in range(N):
        Signals[i,:] = np.random.rand(1000)
    Res = np.zeros(Signals.shape)
    for i in range(N):
        Res[i, :] = energy2(Signals[i, :], window,i)
    print(Res)
    # parallel
    cpu_nb = 11  # multiprocessin
    Res2 = np.zeros(Signals.shape)
    idx = 0
    pool = multiprocessing.Pool(cpu_nb)
    args = []
    for h in range(N):
        args.append([window,Signals[h, :],h])
    A = pool.starmap(energy2, args)
    for imap in range(len(A)):
        Res2[imap, :] = A[imap]


    print(Res - Res2) #find same results

score 1 · Answer 2 · answered Mar 09 '20 at 15:00

That is a lot of code, but I think changing the energy2 function to:

def energy2(items):
    signal, window, i = items
    ...

would do the trick. The problem seems to be that you are passing a list to a function that expects 3 values. Normally you can do this by using a starred expression but in your case it is a bit more delicate than that since you are using map.. So let the function handle its input.

Multiprocessing pool map doesn't accept several arguments as list of lists

2 Answers2