Parallel network requests:Comparing different approaches

Question

I would like to request 100 urls at a time and am currently doing this:

responses = list(PoolExecutor(max_workers=NUM_PARALLEL).map(
                 lambda xml: requests.post(URL, headers=HEADERS, data={'message': xml}), 
                 xmls))

A few questions on this:

Is list the best way to 'evaluate' the actual generator object/expression? Unless I do list I just get something like: <generator object Executor.map.<locals>.result_iterator at 0x10ecf9888>
Is PoolExecutor used frequently to do parallel network requests in python3, or are there are other methods that are more preferable?
What are the differences between PoolExecutor, AsyncIO and concurrent.futures to do something like this?

I think there are examples in the `AsyncIO` docs that are similar to your example. I would probably use `concurrent.futures` - but at times I've thought it would be worthwhile getting more proficient with `AsyncIO`. — wwii, Feb 27 '20 at 19:32
Where does `PoolExecutor` come from? _Is list the best way to 'evaluate'..._ That should be covered in the docs, right? If not, I'm sure there is plenty of information out there. _Is PoolExecutor used frequently to do parallel network requests in python3, or are there are other methods that are more preferable?_ https://meta.stackoverflow.com/questions/261592/how-much-research-effort-is-expected-of-stack-overflow-users — AMC, Feb 27 '20 at 19:40
I forgot to mention that #2 is likely quite subjective. Judging how popular/common a library is can be done to some extent, although it isn't exactly idea. _are there are other methods that are more preferable_ even more so, I would say. — AMC, Feb 27 '20 at 19:53

score 1 · Answer 1 · answered Mar 04 '20 at 22:49

Is list the best way to 'evaluate' the actual generator object/expression? Unless I do list I just get something like: .result_iterator at 0x10ecf9888>

There's a specific answer in Stack Overflow which I think addresses your concern.

Is PoolExecutor used frequently to do parallel network requests in python3, or are there are other methods that are more preferable?

There's other methods, yes, and you can find here in StackOverflow at least one question giving different examples. Within the answers to that question, there's only one person using concurrent.futures.ThreadPoolExecutor and that's the closest you'll get to PoolExecutor there.

What are the differences between PoolExector, AsyncIO and concurrent.futures to do something like this?

The answer to this question can be found in the book Using Asyncio in Python, Chapter 3. Asyncio Walk-Through. More precisely, AsyncIO provides an API that is very similar to the API in the concurrent.futures package. This package provides a ThreadPoolExecutor and a ProcessPoolExecutor. The default is thread-based, but either thread-based or pool-based executors can be used.

Parallel network requests:Comparing different approaches

1 Answers1