Multiprocessing Pool initializer fails pickling

Question

I am trying to use the multiprocessing.Pool to implement a multithread application. To share some variables I am using a Queue as hinted here:

def get_prediction(data):
    #here the real calculation will be performed
    ....


def mainFunction():
    def get_prediction_init(q):
        print("a")
        get_prediction.q = q

    queue = Queue()
    pool = Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])

if __name__== '__main__':
    mainFunction()

This code is running perfectly on a Debian machine, but is not working at all on another Windows 10 device. It fails with the error

AttributeError: Can't pickle local object 'mainFunction.<locals>.get_prediction_init'

I do not really know what exactly is causing the error. How can I solve the problem so that I can run the code on the Windows device as well?

EDIT: The problem is solved if I create the get_predediction_init function on the same level as the mainFunction. It has only failed when I defined it as an inner function. Sorry for the confusion in my post.

Unsure if this is the cause of the issue, but `multiprocessing` code should be shielded by `if __name__ == '__main__':` [in Windows](https://docs.python.org/2/library/multiprocessing.html#windows). — roganjosh, Apr 17 '17 at 18:56
Please post a complete, executable program, as I did for you. When I guess at what the missing pieces are in your edited code, it still works fine for me on Windows. — Tim Peters, Apr 17 '17 at 19:01
@TimPeters can confirm the error from the posted code on Windows 7 at least. Your minimal example works, the posted code throws `PicklingError`. — roganjosh, Apr 17 '17 at 19:05
@TimPeters Sorry (again). I made mistake in my first edit with the copying & pasting of my code. — zimmerrol, Apr 17 '17 at 19:05

Tim Peters · Accepted Answer · 2017-04-17T19:02:24.330

The problem is in something you haven't shown us. For example, it's a mystery where "mainFunction" came from in the AttributeError message you showed.

Here's a complete, executable program based on the fragment you posted. Worked fine for me under Windows 10 just now, under Python 3.6.1 (I'm guessing you're using Python 3 from your print syntax), printing "a" 16 times:

import multiprocessing as mp

def get_prediction(data):
    #here the real calculation will be performed
    pass

def get_prediction_init(q):
    print("a")
    get_prediction.q = q

if __name__ == "__main__":
    queue = mp.Queue()
    pool = mp.Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])
    pool.close()
    pool.join()

Edit

And, based on your edit, this program also works fine for me:

import multiprocessing as mp

def get_prediction(data):
    #here the real calculation will be performed
    pass

def get_prediction_init(q):
    print("a")
    get_prediction.q = q

def mainFunction():
    queue = mp.Queue()
    pool = mp.Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])
    pool.close()
    pool.join()

if __name__ == "__main__":
    mainFunction()

Edit 2

And now you've moved the definition of get_prediction_init() into the body of mainFunction. Now I can see your error :-)

As shown, define the function at module level instead. Trying to pickle local function objects can be a nightmare. Perhaps someone wants to fight with that, but not me ;-)

I edited the post and added more information about the mainFunction. — zimmerrol, Apr 17 '17 at 18:59
Alright! Do you know why it fails on Windows but works on Debian? — zimmerrol, Apr 17 '17 at 19:06
Answering that is a form of "fight with that" ;-) I don't even want to try. Real answers tend to get deep in the weeds, and in the end you have to make the functions global anyway. — Tim Peters, Apr 17 '17 at 19:08
But I'll give the short course: on Debian `Pool` creates worker processes via `fork()` - pickle has nothing to do with that. On Windows `fork()` isn't available, and objects have to be pickled for worker processes to unpickle. On Debian, they're simply inherited from the parent process. — Tim Peters, Apr 17 '17 at 19:13

Multiprocessing Pool initializer fails pickling

1 Answers1

Edit

Edit 2