Why does selenium wait for a long time before executing this code?

Question

I'm trying to do infinite scrolling on this page and here is my code:

from selenium import webdriver
import time

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)

driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
    driver.execute_script("window.scrollTo(0,1000000);")
    time.sleep(2)

So when I run this, it waits for a lot of seconds (more than 1 min sometimes) before doing any scrolling and then waits again for the same amount of time before the next scrolling. The code seems to work fine on other pages. Any ideas on how to fix this?

When I try to use Chrome instead of firefox, I get these errors: driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') added to the .py file.

Traceback (most recent call last):
  File "ok.py", line 8, in <module>
    driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
    'desiredCapabilities': desired_capabilities,
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Answer 1

Switching to Chrome() helped me to solve the problem:

import time
from selenium import webdriver

followers_per_page = 18

driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")

# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count

# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
    driver.execute_script("window.scrollTo(0, 0,1000000);")
    time.sleep(2)

FYI, I'm, using a bit different approach: parsing the count of followers and calculating followers per page taking into account the fact that it loads 18 followers at a time.

I've actually worked on a similar quora question before, see:

• Scrolling web page using selenium python webdriver

---

Well, this was not the first thing coming into my mind. Here's the story.

The problem is that there are pending requests to the http://tch840195.tch.quora.com/up/chan5-8886/updates URL that are taking minutes to complete. This is what makes selenium think the page is not completely loaded. And, things are getting worse - this is a periodical thing that happens every X seconds. Think about it as long pooling.

I've tried multiple things to overcome the problem using Firefox webdriver:

• set webdriver.load.strategy preference to unstable
• set network.http.response.timeout, network.http.connection-timeout and network.http.keep-alive.timeout and network.http.request.max-start-delay preferences

• set page load timeout:

  driver.set_page_load_timeout(3)
  

• set script timeout:

  driver.set_script_timeout(3)
  

• call window.stop(); hoping it would stop active requests:

  driver.execute_script('window.stop();')
  

• updated to most recent Firefox and selenium package versions

One other option that might work is to somehow block the request to that "slow url" either using a proxy server and point firefox to it, or, if it is possible, let Firefox know to blacklist the URL (probably through an extension).

Also see the relevant issue with multiple workarounds inside:

• Support a timeout argument on page load operations

Also see:

• How to Stop the page loading in firefox programmatically?
• Firefox DelayedCommand waits for pending XMLHttpRequests when it doesn't need to.
• FirefoxDriver webdriver.load.strategy unstable findelements getting elements from the wrong page
• Set up a real timeout for loading page in Selenium WebDriver?
• Selenium Firefox Open timeout