Error in use of multiprocessing module with generator using multiple yields

Question

Can some please assist me with the my code below, I'm trying to implement a generator with multiple yields using multiprocessing. I would normally use a next followed by a send for the second and third calls to the generator but I'm not sure how this is done using the multiprocessing module.

from multiprocessing import Pool


def sq_divide(x, y):
    yield x**y
    send_receive = (yield x) #"send_receive" is used as confirmation that text "Done!!" was received

    if send_receive=="Done!!":
        yield x / y


if __name__ == "__main__":
    p = Pool(4)

    #first generator call
    n = p.map(sq_divide, (range(1, 10), range(1, 10)))
    print(n)

    #second generator call
    n = p.map(next, n)

    #last generator call
    n = p.map(n.send, "Done!!")
    print(n)

My expect output is:

first print:
[1, 4, 9, 16, 25, 36, 49, 64, 81]

second print:
[1, 2, 3, 4, 5, 6, 7, 8, 9]

third print:
[1, 1, 1, 1, 1, 1, 1, 1, 1]

I dont think that what you are trying to do can be done like that. Using multiprocessing create a copy of the generator for each process, so it is not really shared. — Netwave, Mar 28 '19 at 10:38
Is there a way it can be implemented that is different from the way I implemented it but yields the same output? @Netwave — Kusi, Mar 28 '19 at 10:47
Well, there are many ways, for example splitting the funtion in two. You need to spificy a bit more. Why do you need it like this? — Netwave, Mar 28 '19 at 10:54
Alright, the code I have written is a condensed version of another. Basically, I need the generator to yield values at different stages. Those values are then used and then the generator must continue from where it left off to the next yield value. However I need all calls to the generator processed quickly cause I have a gif running (using Tkinter) and I don't want it to stop/lag while the generator is processing. — Kusi, Mar 28 '19 at 11:02
Maybe it simpler if you use a thread only for the gif? I think your approach to the problem may be the problem itself. — Netwave, Mar 28 '19 at 11:07
Alright, can one widget in Tkinter be threaded while the others are not?... or do all of them (buttons, labels etc.) need to be threaded? — Kusi, Mar 28 '19 at 11:29
@Kusi: Read [use threads to preventing main event loop from “freezing”](https://stackoverflow.com/a/16747734/7414759) — stovfl, Mar 28 '19 at 11:30

score 1 · Answer 1 · answered Mar 28 '19 at 13:18

You are riding the wrong horse here. Pool workers are supposed to receive some input data to do a job, return their output and process a new task. Here the workers return generator objects which are not pickable hence the error. But more fundamentaly, a generator has to live until it has been exhausted, so it makes no sense in a pool worker process.

Options:

use one single background thread/process to manage all the generators while the main one will handle the UI part
dispatch the generators on a bunch of thread/processes (not from a Pool) with a long lifetime (until all generators have been exhausted)

The choice really depends on the possible bottlenecks suffered by the generators

Error in use of multiprocessing module with generator using multiple yields

1 Answers1