Difference between map() and pool.map()

Question

I have a code like this

def plotFrame(n):
    a = data[n, :]
    do_something_with(a)

data = loadtxt(filename)
ids = data[:,0]  # some numbers from the first column of data
map(plotFrame, ids)

That worked fine for me. Now I want to try replacing map() with pool.map() as follows:

pools = multiprocessing.Pool(processes=1)
pools.map(plotFrame, ids)

But that won't work, saying:

NameError: global name 'data' is not defined

The questions is: What is going on? Why map() does not complain about the data variable that is not passed to the function, but pool.map() does?

EDIT: I' m using Linux.

EDIT 2: Based on @Bill 's second suggestion, I now have the following code:

def plotFrame_v2(line):
    plot_with(line)

if __name__ == "__main__":
    ff = np.loadtxt(filename)
    m = int( max(ff[:,-1]) ) # max id
    l = ff.shape[0]
    nfig = 0
    pool = Pool(processes=1)
    for i in range(0, l/m, 50):
        data = ff[i*m:(i+1)*m, :] # data of one frame contains several ids
        pool.map(plotFrame_v2, data)
        nfig += 1        
        plt.savefig("figs_bot/%.3d.png"%nfig) 
        plt.clf()

That works just as expected. However, now I have another unexpected problem: The produced figures are blank, whereas the above code with map() produces figures with the content of data.

Are you running this on Windows perhaps? When using `multiprocessing` it's best not to rely on globals; especially on Windows the spawned child process may not see changes to globals. — Martijn Pieters, Mar 26 '14 at 13:56
This question may be helpful, don0t know much about this but it can help: http://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments — ederollora, Mar 26 '14 at 13:57

score 4 · Answer 1 · edited May 23 '17 at 12:05

Using multiprocessing.pool, you are spawning individual processes to work with the shared (global) resource data. Typically, you can allow the processes to work with a shared resource in the parent process by make that resource explicitly global. However, it is better practice to explicitly pass all needed resources to the child processes as function arguments. This is required if you are working on Windows. Check out the multiprocessing guidelines here.

So you could try doing

data = loadtxt(filename)

def plotFrame(n):
    global data
    a = data[n, :]
    do_something_with(a)

ids = data[:,0]  # some numbers from the first column of data
pools = multiprocessing.Pool(processes=1)
pools.map(plotFrame, ids)

or even better see this thread about feeding multiple arguments to a function with multiprocessing.pool. A simple way could be

def plotFrameWrapper(args):
    return plotFrame(*args)

def plotFrame(n, data):
    a = data[n, :]
    do_something_with(a)

if __name__ == "__main__":
    from multiprocessing import Pool
    data = loadtxt(filename)
    pools = Pool(1)

    ids = data[:,0]
    pools.map(plotFrameWrapper, zip([data]*len(inds), inds))
    print results

One last thing: since it looks like the only thing you are doing from your example is slicing the array, you can simply slice first then pass the sliced arrays to your function:

def plotFrame(sliced_data):
    do_something_with(sliced_data)

if __name__ == "__main__":
    from multiprocessing import Pool
    data = loadtxt(filename)
    pools = Pool(1)

    ids = data[:,0]
    pools.map(plotFrame, data[ids])
    print results

pools.map(zip([data]*len(inds), inds)) you mean: pools.map(plotFrameWrapper, zip([data]*len(inds), inds)), I guess. This solutions works good. Thanks. However, I don't get your second suggestion. In principle, what I'm doing right now is slicing the data framewise. and then with map() processing the sliced array linewise. — Tengis, Mar 26 '14 at 14:22
Yes, good catch on the `zip` portion. As for the second suggestion, numpy arrays are iterable in the sense that if you say `for x in 2d_array`, each `x` represents one row. So if you slice `data` framewise then feed that into `map`, `map` will iterate through the rows of the sliced array. — wflynny, Mar 26 '14 at 14:35

score 2 · Accepted Answer · edited May 23 '17 at 10:26

To avoid "unexpected" problems, avoid globals.

To reproduce your first code example with builtin map that calls plotFrame:

def plotFrame(n):
    a = data[n, :]
    do_something_with(a)

using multiprocessing.Pool.map, the first thing is to deal with the global data. If do_something_with(a) also uses some global data then it should also be changed.

To see how to pass a numpy array to a child process, see Use numpy array in shared memory for multiprocessing. If you don't need to modify the array then it is even simpler:

import numpy as np

def init(data_): # inherit data
    global data #NOTE: no other globals in the program
    data = data_

def main():
    data = np.loadtxt(filename) 
    ids = data[:,0]  # some numbers from the first column of data
    pool = Pool(initializer=init, initargs=[data])
    pool.map(plotFrame, ids)

if __name__=="__main__":
    main()

All arguments either should be explicitly passed as arguments to plotFrame or inherited via init().

Your second code example tries to manipulate global data again (via plt calls):

import matplotlib.pyplot as plt

#XXX BROKEN, DO NOT USE
pool.map(plotFrame_v2, data)
nfig += 1        
plt.savefig("figs_bot/%.3d.png"%nfig) 
plt.clf()

Unless you draw something in the main process this code saves blank figures. Either plot in the child processes or send data to be plotted to the parent processes explicitly e.g., by returning it from plotFrame and using pool.map() returned value. Here's a code example: how to plot in child processes.

Difference between map() and pool.map()

2 Answers2