problems downloading large files with requests?

Question

I'm trying to download a video file using an API, the equivalent curl command works without problem, the python code below works without error for small videos:

with requests.get("http://username:password@url/Download/", data=data, stream=True) as r:
    r.raise_for_status()
    with open("deliverables/video_output34.mp4", "wb") as f:
        for chunk in r.iter_content(chunk_size=1024):
            f.write(chunk)

it fails for large videos (failed for video ~34M) (the equivalent curl command works for this one)

Traceback (most recent call last):
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/adapters.py", line 479, in send
    r = low_conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/adapters.py", line 482, in send
    r = low_conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/nabil/.local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: Remote end closed connection without response

I've checked links like the following without success

John Doe · Accepted Answer · 2020-08-13T08:59:31.110

Thanks to SilentGhost on IRC#python who pointed out to this suggesting I should upgrade my requests, which solved it(from 2.22.0 to 2.24.0).

upgrading the package is done like this:

pip install requests --upgrade

Another source that may help someone looking at this question is to use pycurl, here is a good starting point: https://github.com/rajatkhanduja/PyCurl-Downloader

or/and you can use --libcurl to your curl command to get a good indication on how to use pycurl

problems downloading large files with requests?

1 Answers1