Unable to use pair of items within a function when I go for concurrent.futures

Question

I'm trying to make use of concurrent.futures within my script below. There are two functions make_requests() and get_info() within it. When make_requests() function yields links and proxy_list, I intend to pass them within get_info() function.

Usually this is how it works:

if __name__ == '__main__':
    # "proxy_lists" is the holder of different proxies
    proxies = [proxy_lists]

    # make_requests() function yields "links" and "proxy_list"
    for item,elem in make_requests(url,proxies):

        # pass "links" and "proxy_list" to get_info() function
        get_info(item,elem)

I don't find any idea to do the same using concurrent.futures as it requires a paired loop to produce two type of elements by this make_requests() function and then pass the two type of elements within get_info() function like I did above:

import concurrent.futures

if __name__ == '__main__':
    proxies = [proxy_lists]

    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        future_to_url = {executor.submit(get_info,link): link for link in make_requests(url,proxies)}
        concurrent.futures.as_completed(future_to_url)

How can I pass pair of items within get_info() function when I use concurrent.futures?

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

I think you just want to do: executor.submit(get_info, *link)

this is because Executor.submit() is documented as:

submit(fn, *args, **kwargs)

Schedules the callable, fn, to be executed as fn(*args **kwargs)

this uses standard arbitrary argument handling to make sure that everything goes through to fn

the *link treats link as a sequence, unpacking it as arguments to submit, which in turn does then same when it eventually invokes fn

Unable to use pair of items within a function when I go for concurrent.futures

1 Answers1