How to make a function behave asynchronously?

Question

I've recently been having a run-up with asynchronous functions in Python, and I wonder how one could make a synchronous function into an asynchronous one.

For example, there is the library for translation via google api pygoogletranslation. One could most possibly wonder, how to translate many different words asynchronously. Of course, you could place it into one request, but then google api would consider it a text and treat it accordingly, which will cause incorrect results.

How could one turn this code:

from pygoogletranslation import Translator
translator = Translator()
translations = []
words = ['partying', 'sightseeing', 'sleeping', 'catering']
for word in words:
    translations.append(translator.translate(word, src='en', dest='es'))
print(translations)

Into this:

from pygoogletranslation import Translator
import asyncio
translator = Translator()
translation_tasks = []
words = ['partying', 'sightseeing', 'sleeping', 'catering']
for word in words:
    asyncio.create_task(translator.translate(word, src='en', dest='es'))
translations = asyncio.run(
    asyncio.gather(translation_tasks, return_exceptions=True) 
)
print(translations)

Considering the function translate doesn't have a built-in async implementation?

my javascript brain searches for Promise.all equivalents: https://stackoverflow.com/questions/34377319/combine-awaitables-like-promise-all. Basically wrap each request to google in an async function, and use the results once the array of async functions has completed. — Matt Wilde, Feb 21 '21 at 07:08
@MattWilde As noted in the answer `asyncio.gather()` is the closest equivalent to `Promise.all`. The key difference lies elsewhere: in JavaScript everything that could ever block is just async by default, which isn't the case in Python. — user4815162342, Feb 21 '21 at 09:35

Tim · Answer 1 · 2021-02-21T07:31:46.643

1

You will have to create an async function and then run it. Though if translate doesn't have built in async support or is blocking, using async will not make it faster. It's probably better to use multithreading/multiprocessing as suggested in the comments.

async def main():
    async def one_iteration(word):
        output.append(translator.translate(word, src='en', dest='es'))
    coros = [one_iteration(word) for word in words]
    await asyncio.gather(*coros)
asyncio.run(main())

edited Feb 21 '21 at 07:31

answered Feb 21 '21 at 07:24

Tim

3,178
1
13
26

Sorry, as a matter of fact the example I gave is just for showing what I would like to do with any kind of function. I'm just wondering, how to implement an async function, how it works on the inside and why one function can be async and another cannot. – winwin Feb 21 '21 at 07:44
1

Async functions need to be defined with `async def` and it's essentially a nonblocking function that can let other functions run while it waits for results. So if you put some blocking call in an async function, it will still block other functions from running. An simple example is that in async functions you want to use `async.sleep` instead of `time.sleep` since the async version is non-blocking. See the python doc or https://realpython.com/async-io-python/ for more details. – Tim Feb 21 '21 at 07:51

score 1 · Answer 2 · answered Feb 21 '21 at 15:42

As mentioned in other answers, calling a blocking function is useless with ayncio. In this particular case, I suggest you use google-cloud-translate, which is the official translate library from Google.

You could have done something like this in your current library:

async def do_task(word):
    return translator.translate(word, ...)

def main():
    # Create translator
    ...
    asyncio.gather(do_task(word) for word in [])

But this will just run the task in the same way without asyncio. The real gain in asyncio is that, when is something pending or waiting, it can do something else. eg, while waiting for response from server, it can send another request.

How will Python know that some work is pending? Only when the function (coroutine here) notifies the event loop via await keyword. So you definitely need to use a library that natively supports async operations. The above mentioned google-cloud-translate is such a library. You can do:

from google.cloud import translate


async def main():
    # Async-supported google translator client
    client = translate.TranslationServiceAsyncClient()
    words = ['partying', 'sightseeing', 'sleeping', 'catering']
    results = await asyncio.gather(*[client.translate_text(parent=f"projects/{project_name}", contents=[word], source_language_code="en", target_language_code="es") for word in words])
    print(results)

asyncio.run(main())

You can see that this client actually takes list of strings as input, so you could directly pass the list of strings here. According to docs, the limit for that is 1024. So if your list is bigger, you have to use this for loop.

You might have to set up credentials etc for this client though, which is outside the scope of this question.

score 0 · Answer 3 · answered Feb 21 '21 at 09:38

To make a function async, you need to define it with async def and change it to use other async functions for anything that might block - for example, instead of requests you'd use aiohttp, and so on. The point of the effort is that the function can then be executed by an event loop along with other such functions. Whenever an async function needs to wait for something, as signaled by the await keyword, it suspends to the event loop and gives others a chance to execute. The event loop will seamlessly coordinate concurrent execution of a possibly large number of such async functions. See e.g. this answer for more details.

If a critical blocking function that you are depending on doesn't have an async implementation, you can use run_in_executor (or, beginning with Python 3.9, asyncio.to_thread) to make it async. Note, however, that such solutions are "cheating" because they use threads under the hood, so they will not provide benefits normally associated by asyncio such as ability to scale beyond the number of threads in the thread pool, or ability to cancel execution of coroutines.

How to make a function behave asynchronously?

3 Answers3