python url request to finance website hangs

Question

I am struggling to get reliable dividend data for my website so I have the following script to hit nasdaq.com:

import requests

dividends = "https://www.nasdaq.com/market-activity/stocks/hd/dividend-history"

response = requests.get(dividends)

The script just hangs and does nothing, the traceback when you cancel it looks like this:

^CTraceback (most recent call last):
  File "/home/cchilders/projects/stocks_backend/scripts/get_dividends_nasdaq_dot_com.py", line 5, in <module>
    response = requests.get(dividends)
  File "/home/cchilders/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/cchilders/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/cchilders/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/cchilders/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/cchilders/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1273, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1129, in read
    return self._sslobj.read(len, buffer)
KeyboardInterrupt

The test script works and shows 200 response:

response = requests.get("https://www.google.com")
print(response)

does this mean that the site has blocked requests module and other libraries from connecting or is there something else I can do? I cannot find dividends data going back more than a few years from any site except this one

score 3 · Accepted Answer · answered Sep 16 '22 at 09:22

The site has blocked some user-agents. Try using the user-agent of a browser:

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0'
}

dividends = "https://www.nasdaq.com/market-activity/stocks/hd/dividend-history"

response = requests.get(dividends, headers=headers)

print(response)

Result:

<Response [200]>

python url request to finance website hangs

1 Answers1