python stderr gets appended on top while redirecting with bash

Question

I have the following problem:

def main():

    try: 
        # DO some stuff here, load some class and execute it thats all
        processFile = cf.jsonLoad(opts.processFile);
        # load process module/class
        mod, ProcessClass = iH.importClassFromModule( **processFile["processClass"] )
        process = ProcessClass( processFile, jobGenModules = {"importHelpers":iH, "commonFunctions" : cf} )
        process.doProcessing()

    except Exception as e:
        print("====================================================================", file=sys.stderr)
        print("Exception occured here: " + str(e), file=sys.stderr)
        print("====================================================================", file=sys.stderr)
        traceback.print_exc(file=sys.stderr)
        return 1

if __name__ == "__main__":
   sys.exit(main());

when I launch the program from bash by redirecting stderr and stdout to the file by doing

yell() { echo "$0: $*" >&2; }
die() { yell "$*"; cleanup ; exit 111 ; }
try() { "$@" || die "cannot $*"; }
executeProg(){
    python3 main.py >> log.txt 2>&1
}
try executeProg

the stderr output of the exception above gets appended at the beginning of the file?

Is that general behavior of bash? Or am I doing something wrong in python? How can I make that the stderr is also appended at the position where the stdout is, meaning at the end of the file...?

ok this is probably a buffering problem, stdcout is not always buffered the same as stderr — Gabriel, Oct 23 '15 at 09:04

score 2 · Accepted Answer · edited May 23 '17 at 11:44

Such behaviour is typically caused by buffering, e.g. the stdout output of your python code "DO STUFF HERE" has not yet been sent to the file. Typically you will not see any problem in the console, but the "problem" appears once you redirect to a file.

You might try to manually flush stdout before outputting the exception details, as follows:

except Exception as e:
    sys.stdout.flush()
    print(...

You could also try disabling buffering altogether, see Disable output buffering

shanmuga · Answer 2 · 2018-02-17T12:50:22.880

1

2>&1 This basically says redirect stderr to stdout.

I tried the following code.

a = 0
b = 1

print("hi")
try:
    c = b/a
except ZeroDivisionError:
    print("ZeroDivisionError")
print("done")

To execute I used python a.py >> a.txt 2>&1. The content of a.txt after this is

hi
ZeroDivisionError
done

This shows whatever is printed first comes first in the file.

edited Feb 17 '18 at 12:50

answered Oct 23 '15 at 08:44

shanmuga

4,329
2
21
35

correct, and this means it does not say anything about the filestreams beeing in sync? (is that even possible?) – Gabriel Oct 23 '15 at 08:45
I tried the an example and the exception is below what ever output was already printed before the exception. Whatever comes first is written to file first. – shanmuga Oct 23 '15 at 08:49

python stderr gets appended on top while redirecting with bash

2 Answers2