Why is Python raising exception, despite a bare Except block?

Question

Imagine the following code:

def query(sql, data):
    with conn as cursor:
        try:
            cursor.execute(sql, data)
            rows = cursor.fetchall()
            conn.commit()
        except Exception as e:
            print(cursor._last_executed)
            print(e)

When calling it, an err.InterfaceError("(0, '')") is risen from the last line: print(e).

I could even understand it if it was risen from print(cursor._last_executed), as cursor might be not available or something. But this is not the case.

Why, when my bare exception is supposed to handle everything?

Please ignore the discussion whether bare exceptions are a good or bad practice, this is another topic. The question is how come the exception is risen at all in this case.

Edit: the exception is risen very rarely, under heavy DB load. You will not be able to reproduce it.

Edit2: I managed to copy the traceback as text from the sentry report:

InterfaceError: (0, '')
  File "run_signal_generator.py", line 39, in <module>
    main()
  File "run_signal_generator.py", line 35, in main
    ds.run_trades_stream()
  File "/home/deribit/rubber-band/data_stream/data_streamer.py", line 223, in run_trades_stream
    self.process_data(data)
  File "/home/deribit/rubber-band/data_stream/data_streamer.py", line 97, in process_data
    self.start_new_candle(timeframe)
  File "/home/deribit/rubber-band/data_stream/data_streamer.py", line 117, in start_new_candle
    self.notify()
  File "/home/deribit/rubber-band/data_stream/observer.py", line 13, in notify
    observer.update()
  File "/home/deribit/rubber-band/data_stream/observer.py", line 26, in update
    self.process_data()
  File "/home/deribit/rubber-band/data_stream/signal_generator.py", line 131, in process_data
    return self.process_potential_new_trade()
  File "/home/deribit/rubber-band/data_stream/signal_generator.py", line 160, in process_potential_new_trade
    return self.process_enter_signal()
  File "/home/deribit/rubber-band/data_stream/signal_generator.py", line 407, in process_enter_signal
    trade_id = self.store_enter_signal_db(data)
  File "/home/deribit/rubber-band/data_stream/signal_generator.py", line 522, in store_enter_signal_db
    return query(sql, db_data)["id"]
  File "/home/deribit/rubber-band/helpers/mysql.py", line 19, in query
    print(e)
  File "pymysql/connections.py", line 881, in __exit__
    self.commit()
  File "pymysql/connections.py", line 798, in commit
    self._execute_command(COMMAND.COM_QUERY, "COMMIT")
  File "pymysql/connections.py", line 1122, in _execute_command
    raise err.InterfaceError("(0, '')")

Indeed it claims the exception is risen from this line.

Your bare exception block *is* handling everything. It's merely that while handling it, another exception is being raised. The relevant discussion here would be why the exception is raising an exception while trying to print it. — deceze, Jan 22 '20 at 09:16
Are you sure the error is _raised_ at `print(e)`, and not simply _printed_ there? — Amadan, Jan 22 '20 at 09:18
There is no common traceback available. I have a sentry traceback which I uploaded as an image file and got downvotes and advice to remove it. According to the sentry traceback, the error is indeed risen from this line. My only explanation would be that the sentry traceback is wrong and it is risen from the above line. — Nikolay Shindarov, Jan 22 '20 at 09:25
OK I figured the cause - the exception was indeed risen from the ```print``` line, if you are curious you can check my answer. — Nikolay Shindarov, Jan 22 '20 at 10:54

score 3 · Answer 1 · answered Jan 22 '20 at 09:20

3

Seems similar to this old question: mysql connections and cursors are not thread-safe so if you're sharing connections between multiple threads (instead of having one connection per thread) their state can break at essentially any moment.

It's possible that for some reason __str__ on MySQL's connection errors ping the server to try and get more information, which break if e.g. an other thread has already changed the connection state. cursor._last_executed might not have that issue due to just saving the information locally either when executing the request or when the response comes back.

And as other commenters noted, if you get an exception provide the entire traceback and error message not a small fraction of the information. If you need to anonymise part of the traceback do so but the less ESP or assumptions are necessary the better.

answered Jan 22 '20 at 09:20

Masklinn

34,759
3
38
57

I know this might occur due to multiple threads, it's not the case here. It happens in a single thread, under very heavy DB load. I guess, a connection timeout happens or something so I can understand why the error occurs. What I don't understand is why it is risen. If the exception was risen from the ```cursor._last_executed``` line, that makes sense. But from the sentry report I have, it is risen from the below line, ```print(e)```. So my only guess at this point would be the sentry report is not accurate and it was indeed risen from ```cursor._last_executed```. – Nikolay Shindarov Jan 22 '20 at 09:27
1

thanks for your reply, I appreciated it and upvoted it. Eventually I figured the cause, if you are curious you can check my answer. – Nikolay Shindarov Jan 22 '20 at 10:53
Ah right, so the issue is that the transaction was in a failed state but since it was in the contextmanager and you'd *suppressed* the exception without cleanup, the context manager tried to commit the transaction, which was in an invalid state, which blows up. And it'd "point to" the print because that's the last line of the context manager and it doesn't really have a better line to show (a common issue for exceptions triggered by `__exit__`). – Masklinn Jan 22 '20 at 11:05

Nikolay Shindarov · Accepted Answer · 2020-01-22T11:02:47.827

1

Figured it.

This is what is going on:

1) The main query raise an error.

2) It is handled.

3) As the query is inside a context manager, the context manager exites it's object, e.g. connection instance.

4) As the MySQL server has gone away, the __exit__ method of the connection class can not execute properly.

5) As it can not execute properly, it raises an error within the scope of the context manager, outside the scope of the bare handling exception inside query.

You can get the same result if you experiment with the following code:

class ContextManagerException(Exception):
    pass


class TryBlockException(Exception):
    pass


class A(object):
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, tb):
        raise ContextManagerException


with A() as a:
    try:
        raise TryBlockException
        pass
    except Exception as e:
        print(e)

As you can see, the risen exception is indeed risen from the context manager, and indeed it is risen from the print(e) line, as it is the last line to execute from the context manager.

edited Jan 22 '20 at 11:02

answered Jan 22 '20 at 10:52

Nikolay Shindarov

1,616
2
18
25

Wrt (4), it's probably less "the mysql server has gone away" and more "the transaction is in a failure state" (which also triggered the python exception), and the only thing you can do on a transaction in a failure state is to ROLLBACK. The context manager would be set up to either COMMIT in the normal case or ROLLBACK in case of exception, but since you've caught the exception and not re-raised it, it thinks all's gone well and tries to commit, which fails. The exception it triggers (and the error message) suck though. – Masklinn Jan 22 '20 at 11:11
More precisely, (4) would be connection timeout IMO. But anyway, if I really insist to handle the exception I can put the context manager in another try... except block and it will be fine. – Nikolay Shindarov Jan 22 '20 at 11:24
You really should not do that. Either use a context manager and let it handle the transactional lifecycle or don't and handle the entire thing yourself. Especially since you're already committing by hand instead of letting the context manager do that. Note that you can print your information and still use the context manager correctly if you just re-raise it (a bare `raise` in your exception handler). – Masklinn Jan 22 '20 at 11:47
Sorry, I didn't understand. What is so bad about handling the CM error and what bad consequences it may have? – Nikolay Shindarov Jan 22 '20 at 11:54
You're duplicating work and putting the context manager in an unexpected state is all. The entire point of the context manager here is to automatically open a transaction then either commit or rollback it when the body ends depending whether an exception is raised or not. – Masklinn Jan 22 '20 at 11:56

Why is Python raising exception, despite a bare Except block?

2 Answers2