Python: Subprocess.popen stdout: not in real time buffer

Question

I was tring for so long to catch the stdout of a python scrypt in subprocess in real time.

SOLUTION

main.py

import subprocess
import io
import time
import sys

if __name__ == '__main__':

    command = ["python", "-u", "createValue.py"] # "-u" changed my life

    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)

    for line in iter(p.stdout.readline, b''):
        print(">>> " + str(line.rstrip()))

createValue.py


import time
import sys

i = 0

while i < 5:

    print("I like pizza "+ str(i))
    # sys.stdout.flush() # This is an another solution
    time.sleep(1.5)

    i += 1

Why every solution on Internet work without "-u" for them but not on my PC ? I use Python 3.6.5

AFAIK, -u means unbuffered which translates to: Python will not buffer the output, it will send back the data as soon as it is pointed to stdout. Isn't that what you want? I don't see any issues with your code. — Bruno, Jun 18 '21 at 15:05
Yes, does-it work in real time without "-u" for you ? Every solution on Internet doesn't use "-u" Exemple 1: https://stackoverflow.com/questions/1606795/catching-stdout-in-realtime-from-subprocess Exemple 2: https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command — Syl, Jun 18 '21 at 15:17

score 2 · Accepted Answer · answered Jun 18 '21 at 16:05

Yes, it's because you need to add a environment variable called PYTHONUNBUFFERED and set it to = 1

"...Setting PYTHONUNBUFFERED to a non empty value ensures that the python output is sent straight to terminal..."

Check the solution below using your code:

import os # Added this line
import subprocess
import io
import time
import sys

if __name__ == '__main__':
    os.environ["PYTHONUNBUFFERED"] = "1" # Added
    command = ["python", "createValue.py"] # Removed "-u"

    p = subprocess.Popen(command,  
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    print("init")
    for line in iter(p.stdout.readline, b''):
        print(">>> " + str(line.rstrip()))

Python: Subprocess.popen stdout: not in real time buffer

1 Answers1