python buffering in sys.stdout

Question

Minimal example:

def main():
    for i in range(100):
        print("One line every 2s", end = "\n")
        time.sleep(2)
if __name__ == '__main__':
    with open("log.out", 'w') as out, open("log.err", 'w') as err:
        sys.stdout = out
        sys.stderr = err
        main()

I want the print statements to be written toe the stdout file after every line. I tried unsuccesfully:

python -u thisfile.py

In the bashrc (zshrc)


PYTHONUNBUFFERED = 1

open("log.out", 'wb', buffering = 0)

Changing the main function is no an option for the real case. A solution where I run the python file with bash and redirect errors and output is ok tough.

I know theres lots of questions like this, but none seem to work for me.

The solution should work ideally for python 3.8 and anything newer.

Setting `sys.stdout` to a different file handle does not mean the new file handle inherits whatever buffering the original standard output used. — chepner, Feb 03 '23 at 20:01
Ideally I don't want to change all print statements scattered across some files, but I guess it wil be my last resort — dave, Feb 03 '23 at 20:04
Ideally I just want to change the code in the if __name__ == "main" part — dave, Feb 03 '23 at 20:05

Ahmed AEK · Answer 1 · 2023-02-03T20:13:47.517

you could have a wrapper that automatically flushes stdout after every write. wrapper taken from this answer

import time
import sys

class Unbuffered(object):
    def __init__(self, stream):
        self.stream = stream

    def write(self, data):
        self.stream.write(data)
        self.stream.flush()

    def writelines(self, datas):
        self.stream.writelines(datas)
        self.stream.flush()

    def __getattr__(self, attr):
        return getattr(self.stream, attr)


def main():
    for i in range(100):
        print("One line every 2s", end = "\n")
        time.sleep(2)
if __name__ == '__main__':
    with open("log.out", 'w') as out, open("log.err", 'w') as err:
        sys.stdout = Unbuffered(out)
        sys.stderr = Unbuffered(err)
        main()

I already tries this. I get ValueError: can't have unbuffered text I/O — dave, Feb 03 '23 at 20:07

score 0 · Accepted Answer · answered Feb 03 '23 at 20:06

0

You can use the subprocess module to run the script with the desired options and redirect the output and error streams to the log files:

import subprocess

with open("log.out", "w") as out, open("log.err", "w") as err:
    subprocess.run(["python", "-u", "thisfile.py"], stdout=out, stderr=err)

This will run the script in unbuffered mode (-u option) and redirect the output and error streams to the specified log files.

answered Feb 03 '23 at 20:06

GIZ

4,409
1
24
43

It works! Thanks. Any idea why running the file with python -u from the shell does not behave in the same way? – dave Feb 03 '23 at 20:15
When you run the script directly from the shell using `python -u thisfile.py`, the standard output and error streams are connected to the terminal, so the data is immediately visible. However, when you redirect the standard output and error streams using the `>` operator or by using the `subprocess` module, the streams are connected to files instead of the terminal, and the data is buffered before it's written to the files. This can result in delayed or buffered output. – GIZ Feb 03 '23 at 20:18

score 0 · Answer 3 · answered Feb 03 '23 at 20:12

If you use bash you can use:

[...]$ python -u main.py > >(tee -a log.out) 2> >(tee -a log.err >&2)

main.py

import sys
import time

def main():
    for i in range(100):
        print("One line every 2s", end = "\n")
        time.sleep(2)

if __name__ == '__main__':
    main()

Explanation: How do I write standard error to a file while using "tee" with a pipe?

python buffering in sys.stdout

3 Answers3