Async function call in python

Question

In python I want to create an async method in a class that create a thread without blocking the main thread. When the new thread finish, I return a value from that function/thread.

For example the class is used for retrieve some information from web pages. I want run parallel processing in a function that download the page and return a object.

class WebDown:
    def display(self, url):
        print 'display(): ' + content

    def download(self, url):
        thread = Thread(target=self.get_info)
        # thread join
        print 'download(): ' + content
        # return the info

    def get_info(self, url):
        # download page
        # retrieve info
        return info

if __name__ == '__main__':
    wd = WebDown()
    ret = wd.download('http://...')
    wd.display('http://...')

I this example, in order I call download() for retrieve the info, after display() for print others information. The print output should be

display(): foo, bar, ....
download(): blue, red, ....

yes, but I don't know how to use threading module for call async method — NaN, Oct 26 '14 at 20:03
yes, call another method async with thread and when the thread is completed, return the value — NaN, Oct 26 '14 at 20:07
@NaN Did you try Googling "python threading example? There are thousands of examples that would show you how to do this. [Here's one from SO](http://stackoverflow.com/questions/2846653/python-multithreading-for-dummies), for example. — dano, Oct 26 '14 at 20:26
Look into using Twisted, it has examples for asynchronously fetching web pages. — PaulMcG, Oct 26 '14 at 20:36

score 1 · Answer 1 · answered Oct 26 '14 at 21:28

One way to write asynchronous, non blocking code in python involves using Python's Twisted. Twisted does not rely on multithreading but uses multiprocessing instead. It gives you convenient way to create Deferred objects, adding callbacks and errbacks to them. The example you give would look like this in Twisted, I'm using treq (Twisted Requests) library which makes generating requests a little quicker and easier:

from treq import get
from twisted.internet import reactor

class WebAsync(object):
    def download(self, url):
        request = get(url)
        request.addCallback(self.deliver_body)

    def deliver_body(self, response):
        deferred = response.text()
        deferred.addCallback(self.display)
        return deferred

    def display(self, response_body):
        print response_body
        reactor.stop()

if __name__ == "__main__":
    web_client = WebAsync()
    web_client.download("http://httpbin.org/html")
    reactor.run()

Both 'download' and 'deliver_body' methods return deferreds, you add callbacks to them that are going to be executed when results is available.

score 0 · Answer 2 · answered Oct 26 '14 at 20:54

0

I would simply use request and gevent called grequests.

import grequests
>>> urls = [
    'http://...',
    'http://...'
]
>>> rs = (grequests.get(u) for u in urls)
>>> grequests.map(rs)
[<Response [200]>, <Response [200]>]

answered Oct 26 '14 at 20:54

wenzul

3,948
2
21
33

Async function call in python

2 Answers2