0

Basically, I have got a for loop running however, a few of the iterations seem to take too long. Is there any way I can check how long every iteration takes and then skip it if it takes more than a certain amount of time?

for: if iteration > 1 minute skip

I am trying to identify the content language of a set of URLs in a CSV file. However, some of them seem to be taking too long and therefore I am unable to get an output for a large number of URLs. I tried them in batches of 10 but some of them seem to be taking too long and do not give an output.

for i in TC["Home Page"]:
    try:
        html = requests.get(i).content
        soup = BeautifulSoup(html, 'html.parser')
        Language.append(soup.html["lang"])
    except:
        Language.append("error")
TC["Language"] = Language
TC
Hanuman Rajagopal
  • 21
  • 1

1 Answers1

0

I think the problem is in the response from the site. You could try using the request timeout.

for i in TC["Home Page"]:
    try:
        html = requests.get(i, timeout=0.001).content
        soup = BeautifulSoup(html, 'html.parser')
        Language.append(soup.html["lang"])
    except:
        Language.append("error")
    TC["Language"] = Language
    TC

https://requests.readthedocs.io/en/latest/user/quickstart/#timeouts

Cristian Festi
  • 159
  • 4