Multithreading Python Requests Through Tor

Question

The following code is my attempt at doing python requests through tor, this works fine, however I am interested in adding multithreading to this.

So I would like to simultaneously do about 10 different requests and process their outputs. What is the simplest and most efficient way to do this?

def onionrequest(url, onionid):
    onionid = onionid
    session = requests.session()
    session.proxies = {}
    session.proxies['http'] = 'socks5h://localhost:9050'
    session.proxies['https'] = 'socks5h://localhost:9050'
    #r = session.get('http://google.com')


    onionurlforrequest = "http://" + url

    try:
        r = session.get(onionurlforrequest, timeout=15)
    except:
        return None
    if r.status_code = 200:
        listofallonions.append(url)

Answer 1

I would recommend using the the following packages to achieve this: asyncio, aiohttp, aiohttp_socks

example code:

import asyncio
import aiohttp
from aiohttp_socks import ProxyConnector

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main(urls):
    tasks = []
    connector = ProxyConnector.from_url('socks5://localhost:9150', rdns=True)
    async with aiohttp.ClientSession(connector=connector, rdns=True) as session:
        for url in urls:
            tasks.append(fetch(session, url))
        htmls = await asyncio.gather(*tasks)
        for html in htmls:
            print(html)

if __name__ == '__main__':
    urls = [
            'http://python.org',
            'https://google.com',
            ...
        ]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(urls))

Using asyncio can get a bit daunting at first, so you might need to practice for a while before you get the hang of it.

If you want a more in-depth explanation of the difference between synchronous and asynchronous, check out this question.