Python multiprocessing continuous processing with await

Question

I am using an event based system using the new Python 3.5 coroutines and await. I register events and these events are called by the system.

@event
aysnc def handleevent(args):
    # handle the event

I need to initialize some classes to handle the work(time consuming). Then call instance methods, also time consuming (they actually use selenium to browse certain sites).

Ideally I would want something like the following code

# supposedly since this is multiprocessing this is a different driver per process
driver = None
def init():
    # do the heavy initialization here
    global driver
    driver = webdriver.Chrome()

def longworkmethod():
    ## need to return some data
    return driver.dolongwork()

class Drivers:
""" A class to handle async and multiprocessing"""
    def __init__(self, numberOfDrivers):
        self.pool = multiprocessing.Pool(processes=numberOfDrivers, initializer=init)       

    async def dowork(self, args):
        return self.pool.apply_async(longworkmethod, args=args)


### my main python class
drivers = Drivers(5)

@event
aysnc def handleevent(args):
    await drivers.dowork(args)

@event
aysnc def quit(args):
    ## do cleanup on drivers
    sys.exit(0)

This code doesn't work, but I have tried many different ways and none seem to be able to do what I want.

It doesn't have to be this exact form, but how do I go about mixing the await and coroutines with a program that needs multiprocessing?

score 2 · Accepted Answer · edited May 23 '17 at 12:25

2

While there nothing technically speaking that would limit you from mixing asyncio and multiprocessing, I would suggest avoiding doing so. It's going to add a lot of complexity as you'll end up needing an event loop per thread and passing information back and forth will be tricky. Just use one or the other.

asyncio supplies functions for running tasks in another thread - such as AbstractEventLoop.run_in_executor. Take a look at these answers

https://stackoverflow.com/a/33025287/66349 (calling selenium within a coroutine)
https://stackoverflow.com/a/28492261/66349

Alternatively you could just use multiprocessing as selenium has a blocking (non asyncio) interface, however it sounds like some of your code is using already using asyncio so maybe stick with the above.

edited May 23 '17 at 12:25

Community

1
1

answered Mar 28 '17 at 22:33

Peter Gibson

19,086
7
60
64

I can't avoid the mixing because I need selenium(multiprocessing), and the event system I'm using is strictly asyncio. But I'll check those links out. See if I can make them work. – leftsync Mar 28 '17 at 22:38
selenium is blocking but that doesn't mean you need multiprocessing. Just use `run_in_executor` for the blocking bits and keep everything else in the main thread/event loop. – Peter Gibson Mar 28 '17 at 22:40
So those links seem to help. Should I be use run_in_executor for individual expensive calls like "get", or can I do large expensive functions with multiple expensive calls? Would there be a difference? – leftsync Mar 29 '17 at 16:06
@leftsync It depends on the structure of your calls. Can they all be run at once or do they need to be run in sequence? How many threads can you feasibly support? Is selenium thread safe? – Peter Gibson Mar 30 '17 at 21:56

Python multiprocessing continuous processing with await

1 Answers1