Is it possible for too many nested try/except to cause stack overflow?

Question

I have a mixed dataset which I am attempting to handle all cases by using try/except blocks. Essentially I try the first case, if that fails then I try the next case in the except block, and so on.

It was working OK until I got to 10 levels of except. Is this a thing in Python.

I cannot find a definitive answer to this question. I suspect I am writing really bad code.

I give actual code in the first try block only:


try:
    logf.write(f'Processing Type 8...\n')
    with open(filename, 'r') as las_file:
        l = las_file.readlines()

        # drop the header lines and start reading the file from the header row (indicated by ~A)
        lasdata = [x.replace('\n', '').split() for x in list(
            dropwhile(lambda x: '~A' not in x, l))]

        # delete ~A from the header row
        del lasdata[0][0]

        las_df = pd.DataFrame(lasdata)   

        # now set column names as header
        new_header = [
            'DEPTH', 
            'GR',
            'CALI',
            'BRD',
            'LSD',
            'SSD',
            ]
        las_df = las_df[1:]
        las_df.columns = new_header

        # add hole id as column
        las_df['BHID'] = hole
        las_df['filename'] = filename

        # make a copy and drop unwanted columns
        las_df = las_df[[
            'filename',
            'BHID', 
            'DEPTH', 
            'LSD', 
            'SSD', 
            'GR', 
            'CALI']].copy()

        # append each hole in loop
        type8_df = type8_df.append(las_df)

        # write success to log file
        logf.write(f'Type 8: Hole {hole} Processed OK\n')

    # output to CSV file without the pandas index column
    type8_df.to_csv(path_or_buf='.\\type_8.csv', index=False)
except KeyError as e:
    try:
        do_something()
    except KeyError as e:
        try:
            do_something()
        except KeyError as e:
            try:
                do_something()
            except KeyError as e:
                try:
                    do_something()
            except Exception as e:
                logfile.write(e)

and so on - ten levels deep

This is the error message:

Fatal Python error: XXX block stack overflow Current thread 0x000027a8 (most recent call first): File "e:/python/geology/las_data/las_converter.py", line 398 in las_converter File "e:/python/geology/las_data/las_converter.py", line 471 in

I have more than ten KeyError cases to handle. how do I do this?

UPDATE

I have refactored this where now I generate a list of all of the header/column name cases I have to deal with - when processing some directories of files I get up to 50 cases so my try/except approach was never going to work. I then process each file using an if statement to match by type then process.

Thanks all I learned a lot.

"I suspect I am writing really bad code" This statement might be worth exploring. If it's just keyerrors, you could always translate the code into simple `if` conditions as well. — Paritosh Singh, Aug 28 '19 at 04:34
It is unlikely that just nesting functions inside each other would cause a stack overflow unless there is some sort of recursion. I'd check to make sure that your functions aren't caught in an infinite loop of calling each other. — Mike, Aug 28 '19 at 04:35
it looks like you are missing an except statement at the end. It it is the last one just do except: pass — Brandalf, Aug 28 '19 at 04:36
I do have a final except at the end (have added that to my example), it was all working OK until I added another block — Brett McGregor, Aug 28 '19 at 04:39
@ParitoshSingh that is a good suggestion I will try that. I need to rethink the approach... — Brett McGregor, Aug 28 '19 at 04:41
You're just hitting the maximum number of nested blocks allowed by Python's syntax; see https://stackoverflow.com/questions/44972719/why-does-python-have-a-limit-on-the-number-of-static-blocks-that-can-be-nested — Right leg, Aug 28 '19 at 09:31

Dan D. · Accepted Answer · 2019-09-07T05:30:25.177

3

The following applies to CPython. This limit may not be present in other implementations.

The error is "Fatal Python error: XXX block stack overflow", not "RuntimeError: maximum recursion depth exceeded". The latter would be a stackoverflow. This error occurs when the block stack already has its limit of blocks and you add one more.

for/while/try/except/finally/with blocks can only be nested to a limited depth in a function. The depth limit is 20.

It should be possible to rewrite the code so that it well below this limit. One way is to break out the inner part of the code into its own function which would have its own limit.

edited Sep 07 '19 at 05:30

answered Aug 28 '19 at 04:58

Dan D.

73,243
15
104
123

this makes sense as I have 2x for outside the 10 try/except. I guess that's how I'm hitting the limit. Will try to incorporate your suggestion. will take me some time. thanks – Brett McGregor Aug 28 '19 at 05:19

jnnnnn · Answer 2 · 2019-08-30T01:47:54.197

The error message you posted has a stack trace that indicates what line of your code caused the error:

File "e:/python/geology/las_data/las_converter.py", line 398 in las_converter

File "e:/python/geology/las_data/las_converter.py", line 471 in

Have a look at these lines; they are probably assuming a key is present when it sometimes isn't.

Instead of assuming, you can check first and prevent the KeyError entirely:

if key1 in my_dictionary:
    do_something()
elif key2 in my_dictionary:
    do_something()
elif key3 in my_dictionary:
    do_something()

and so on.

You could even use a loop:

for key in [key1, key2, key3, key4, key5]:
    if key in my_dictionary:
        do_something()

Is it possible for too many nested try/except to cause stack overflow?

2 Answers2