Is it possible to refresh a session in requests?

Question

I create a session using requests.Session(). For some reason the server side closes this connection, so I have to reconnect. The problem is, this session is used in many places, so I'm wondering is it possible to rebuild a TCP connection but keep the session object so that I can still use it?

Example:

s = requests.Session()

class B:
    def __init__(self, session):
        self._session = session

    def get(self):
        self._session.get('some_url')

b1 = B(s)
b2 = B(s)
b3 = B(s)

# some get calls
...
# then connection is closed

# some get calls
...

If I could keep the seesion object, there's no need to replace every _session in every B instance.

Error log:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/main.py", line 108, in <module>
    main()
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/main.py", line 89, in main
    m.detect_new_question()
  File "/Users/laike9m/ICT/zhihu-analysis/dynamic/monitor.py", line 32, in detect_new_question
    question = latest_question = next(it)
  File "/usr/local/lib/python3.5/site-packages/zhihu/topic.py", line 269, in questions
    res = self._session.get(question_url, params=params)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

This is a very common issue: Python handling socket.error: [Errno 104] Connection reset by peer. I don't have control over the server so I don't know why or how this happens.

The server does support keep-alive cause I'm able to make hundreds of requests(the period lasts for an hour or more).

HTTP connections are *stateless*, so even with Keep-Alive closing is the norm. Are you confusing cookie expiration with connections being closed perhaps? — Martijn Pieters, Jan 12 '16 at 15:09
Unless you're using `HTTP/2.x` which is the new version of the HTTP protocol, what @MartijnPieters says is correct. But it would be interesting to see if Python does implement a "refresh" to the request library to better match `HTTP/2.x`. — Torxed, Jan 12 '16 at 15:11
@Torxed: the `requests` library doesn't support HTTP/2.x. Not yet, at any rate. — Martijn Pieters, Jan 12 '16 at 15:12
@MartijnPieters Good clarification. I simply made my comment for future readers and for those who try to run a web-server in strict mode (not enabling legacy protocols). And stated that it would be interesting in the future to see if Python does implement such a feature considering there is support for it in the protocol in the form of `PING`. I guess the official Python lib in the future will support the protocol in it's fullest form tho so I guess my remark is redundant. — Torxed, Jan 12 '16 at 15:15
@MartijnPieters It's connection being closed not cookie expiration, I'm not confused……I'll edit my question. — laike9m, Jan 12 '16 at 15:16
@laike9m: the session object creates connections as needed from the pool and supports keep-alive. If the server claims to support keep-alive but closes the connections prematurely without communicating this, then the server is broken. — Martijn Pieters, Jan 12 '16 at 15:17
@laike9m Then what Martijn sad is True, the server is supposed to close the connection after each request. That's how HTTP/1.X works. `Open -> Make request -> Get response -> Close` in almost every case. I suggest you create a function `def getURL(url)` that uses the `requests` lib, and does the session thing for you. — Torxed, Jan 12 '16 at 15:17
@Torxed: with keep-alive you can serially re-use the connection for more requests. The server is still supposed to close cleanly, IIRC. — Martijn Pieters, Jan 12 '16 at 15:18
@MartijnPieters You might be right, I won't argue for the sake of arguing but most servers claiming to support the `Connection: keep-alive` header object won't actually honor it. I have yet to understand why this is the case but in theory yes you should be able to serially re-use the connection. But in most cases, you'll need to re-initiate the connection and such a flag is more used to proxies and firewalls to keep track of the connections, IIRC (which i probably don't..) — Torxed, Jan 12 '16 at 15:20
@Torxed http://docs.python-requests.org/en/latest/user/advanced/#keep-alive, `thanks to urllib3, keep-alive is 100% automatic within a session!`. — laike9m, Jan 12 '16 at 15:24
@laike9m Yepp, note that this is if the server honors it. I was just about to post a piece of code that _might_ help you. Again, this is if the server actually honors what it says it supports. This header object is rarely usable by a client or vice versa, I've developed a few HTTP Server scripts over the years and the some what wild interpretation of the RFC standard among client software is baffling at best of times. Here's a gist that you'd might be able to use: https://gist.github.com/Torxed/3e1d12ba00dd9358c210 (again, I strongly recommend writing a class/function to issue requests) — Torxed, Jan 12 '16 at 15:32
@Torxed Thank you. The situation is, I know the server honors it. I'm able too make many(hundreds) requests via the same session, but it also accidentally closes it. — laike9m, Jan 12 '16 at 15:36
@MartijnPieters What information do you think my question lacks? I'll add them if you let me know. — laike9m, Jan 12 '16 at 15:37
@laike9m If you're talking about an anomaly where the server does something it normally doesn't. Then this is a exception case where you should handle the exception appropriately. Normally with a class/function that can pick this anomaly up and correct it, in this case re-create the session and carry on as it usually would. — Torxed, Jan 12 '16 at 15:37
@Torxed It is an exception and I will handle it as an exception. However the problem, as I said, still lies in replacing the stale session object in every `B` instance. I'm trying to find a convenient way to do this. — laike9m, Jan 12 '16 at 15:40
@laike9m Your issue is not that the connection is being reset, your issue is that you get `TypeError: getresponse() got an unexpected keyword argument 'buffering'` which is something completely different. — Torxed, Jan 12 '16 at 16:07
@laike9m: just keep using the session in that case. Retry the same request, a new connection will be opened for that. — Martijn Pieters, Jan 12 '16 at 16:07
@laike9m: in other words: there is *no need to replace the session* for that case. It is not stale, it just lost a connection from the pool. It'll reopen another one as needed. — Martijn Pieters, Jan 12 '16 at 16:08
@Torxed: no, that's a side effect of using Python and not suppressing the last, already handled exception in the same function (where there is some code to handle backwards compatibility, I think). The connection is still being reset. — Martijn Pieters, Jan 12 '16 at 16:09
@MartijnPieters Ah my bad, heh.. All these years of programming and I still mix up the order of which the stack traces come. My apologies. — Torxed, Jan 12 '16 at 16:10
@MartijnPieters Thx, I'll try if it works that way. I'll accept your answer if it does. — laike9m, Jan 12 '16 at 16:14

Martijn Pieters · Accepted Answer · 2016-01-12T16:17:12.370

Either make your B() a singleton, or make the session a class attribute (and thereby effectively a global).

For example, making it a class attribute only when you create at least one instance could look like this:

class B:
    def __init__(self):
        if not hasattr(type(self), '_session'):
            self._create_session()

    @classmethod
    def _create_session(cls):
        cls._session = requests.Session()

    def get(self):
        self._session.get('some_url')

If using the session can raise an exception because the server is not closing a session connection correctly, just re-create the session at that time:

    def __init__(self):
        if not hasattr(type(self), '_session'):
            self._create_session()

    @classmethod
    def _create_session(cls):
        cls._session = requests.Session()

    def get(self):
        retries = 5
        while retries:
            try:
                return self._session.get('some_url')
            except requests.ConnectionException as e:
                last_connection_exception = e
                retries -= 1
        raise last_connection_exception

The above example retries up to 5 times. You do not need to re-create the session each time. If a connection has been closed, even with an exception, the session object will just create a new TCP/IP connection for the next request.

If you find that the session object is somehow shot and no longer capable of creating new connections, while a fresh session does work, then that'd be a bug. Please report that to the project with a suitable MCVE.

Yep, I just verified on my own server that the session will be re-created automatically with `requests` lib. There's no need to recreate or even bother monitoring for a broken pipe/session. I do however start to wonder what the OP's actual problem is. — Torxed, Jan 12 '16 at 15:53
@Torxed A common issue: http://stackoverflow.com/questions/1434451/what-does-connection-reset-by-peer-mean. requests can't deal with it but raise an exception. — laike9m, Jan 12 '16 at 15:57
@laike9m This is what we're trying to tell you. The server might not honor the `keep-alive` and thus dropping the TCP session. Other reasons for this might be network spikes, wifi connection being iffy or otherwise non-recoverable fatal errors in the network. Either way, `requests` library will handle these, if not, please state your actual error message and not what you're experiencing. — Torxed, Jan 12 '16 at 15:58

Is it possible to refresh a session in requests?

1 Answers1

Linked