python, iterate on subprocess.Popen() stdout/stderr

Question

There are a lot of similar posts, but I didn't find answer.

On Gnu/Linux, with Python and subprocess module, I use the following code to iterate over the stdout/sdterr of a command launched with subprocess:

class Shell:
    """ 
    run a command and iterate over the stdout/stderr lines
    """

    def __init__(self):

        pass

    def __call__(self,args,cwd='./'):

        p = subprocess.Popen(args,
                cwd=cwd, 
                stdout = subprocess.PIPE,
                stderr = subprocess.STDOUT,
                )

        while True:

            line = p.stdout.readline()
            self.code = p.poll()

            if line == '':
                if self.code != None:
                    break
                else:
                    continue

            yield line

#example of use
args = ["./foo"]
shell = Shell()
for line in shell(args):
     #do something with line
     print line,

This works fine... except if the command executed is python, for example `args = ['python','foo.py'], in which case the output is not flushed but printed only when the command is finished.

Is there a solution?

Answer 1

Check out How to flush output of Python print?.

You need to run the python subprocess with the -u option:

-u Force stdin, stdout and stderr to be totally unbuffered. On sys‐ tems where it matters, also put stdin, stdout and stderr in binary mode. Note that there is internal buffering in xreadlines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop.

Or, if you have control over the python sub-process script you can use sys.stdout.flush() to flush the output every time you print.

import sys
sys.stdout.flush()