6

I was looking into the multiprocessing.Pool for workers, trying to initialize workers with some state. The pool can take a callable, initialize, but it isn't passed a reference to the initialized worker. The few example that I've seen utilize it call global variables, which seems really nasty.

Is there any good way to initialize worker state using multiprocessing.Pool?

Edit: An example:

I have workers, each of which do a bit relatively expensive initialisation (binding to a socket), which I don't want to have to do every time. I could initialize my sockets by hand, then pass them in when I assign work, but sharing file descriptors across processes is complicated, if not impossible. So I would have to initialize and bind every time I wanted to process a request.

stillinbeta
  • 1,977
  • 3
  • 17
  • 23
  • 1
    I'm a little unclear why your initializer function needs a reference to the worker, maybe you could provide a slightly more concrete, but minimal example of what you are trying to do. – mgilson Mar 30 '12 at 14:11
  • You are aware that there's no way to tell *which* worker a particular function call will get assigned to? So why not simply include that state in the function arguments? – Sven Marnach Mar 30 '12 at 14:16

1 Answers1

7

Technically speaking, the right thing to do would be having the result of the initialization function passed as argument to every function executed by the worker.

It's also true that in this context is fine and safe to have global variables, since by construction they result private objects living in the separate domains of different processes.

My general suggestion is to build functions with a sane reentrant programming style, and to allow global variables while exploiting the multiprocessing functionality.

Keeping your example, the following send function requires some context (in this case, a socket):

def send(socket, data):
    pass # ... your code here
    return dust

The initialization code and the base code executed by the worker will rely on global variables for convenience.

socket = None
def init(address, port):
    global socket
    socket = magic(address, port)

def job(data):
    global socket
    assert socket is not None
    return send(socket, data)

pool = multithreading.Pool(N, init, [address, port])
pool.map(job, ['foo', 'bar', 'baz'])

By coding it in this way it gets simple and natural to test it without multiprocessing. You can think of your global state as a perfectly safe context capsule.

As additional point of convenience, keep in mind that multiprocessing is not very good at sending complex data around (e.g. callbacks). The best approach is sending simple pieces of data (strings, lists, dictionaries, collections.namedtuple ...) and reconstruct the complex data structures on the worker side (using the initialization function).

Dacav
  • 13,590
  • 11
  • 60
  • 87
  • This was a major eye-opener for me, thanks! I'm now using this pattern for pypdfium2's parallel renderer implementation. – mara004 Jul 29 '23 at 22:16