0

I am using Selenium with Python to obtain the urls (including img urls for ads which are typically located inside nested iframes) of all the resources requested by a webpage. This however, may not work for resources within iframes and for ajax requests. Reading the documentation on this and looking at other StackOverflow answers, it seems like you can use ExpectedCondition to wait for an element to appear. But in my case, where I have to load several websites (several thousand Top Alexa sites), there may not be any common element on which I could wait for. I have been using driver.set_page_load_timeout(60) for that since 60 seconds seemed like a sufficiently long time for all the nested iframes to eventually turn up. I was wondering if anyone could confirm whether this is the correct way to go about it, and whether there is any other way which is better than this. Thanks.

EDIT: I realize the question has been answered before, but for clarification of the specifics of my question, I believe all those answers offer a solution in terms of detecting an element which you know exists on the page, whereas in my case, I don't have any such knowledge pre-hand.

QPTR
  • 1,620
  • 7
  • 26
  • 47

1 Answers1

3

I use selenium every day, and till my knowledge, there is no way to confirm iframes or ajax loading completely or not, the only thing till now ( in my knowledge) is setting page load time out or time.sleep to the extend that you think it is suitable for the cases that you cant find a common element as an indicator to check if the page load completely or not. Many questions about this already answered Selenium Python: how to wait until the page is loaded?

Community
  • 1
  • 1
Phung Duy Phong
  • 876
  • 6
  • 18
  • Phong Thanks for the answer, I'll try to use the Expected Condition technique. At the moment, the issue I am having is that both `expected_conditions` and `set_page_load_timeout` don't follow the `60s` limit I give them, the first one times out much later, and the second one times out much eariler. Any ideas? – QPTR May 02 '17 at 06:48