Unbuffered subprocess output (last line missing)

Question

I must be overlooking something terribly obvious. I need to execute a C program, display its output in real time and finally parse its last line, which should be straightforward as the last line printed is always the same.

process = subprocess.Popen(args, shell = True,  
                           stdout = subprocess.PIPE, stderr = subprocess.PIPE)

# None indicates that the process hasn't terminated yet.
while process.poll() is None:

    # Always save the last non-emtpy line that was output by the child
    # process, as it will write an empty line when closing its stdout.
    out = process.stdout.readline()
    if out:
        last_non_empty_line = out

    if verbose:
        sys.stdout.write(out)   
        sys.stdout.flush()

# Parse 'out' here...

Once in a while, however, the last line is not printed. The default value for Popens's bufsize is 0, so it is supposed to be unbuffered. I have also tried, to no avail, adding fflush(stdout) to the C code just before exiting, but it seems that there is absolutely no need to flush a stream before exiting a program.

Ideas anyone?

If subprocess generates enough output to fill its OS pipe buffer then the child process hangs. Don't use `PIPE` unless you read from the pipe. You could use `deque(iter(process.stdout.readline, b''), maxlen=1).pop()` to get the last line. Or [to read line by line: `for line in iter(process.stdout.readline, b''):...`](http://stackoverflow.com/a/17698359/4279) — jfs, Sep 21 '14 at 16:12

score 3 · Answer 1 · answered Mar 12 '10 at 12:36

3

The problem is that you are reading lines till the process exits, (process.poll()) while you do use buffering because of the shell flag.

You would have to keep reading process.stdout till you reach the end of the file or the empty line.

answered Mar 12 '10 at 12:36

Wolph

78,177
11
137
148

Douglas Leeder · Accepted Answer · 2010-03-12T14:32:01.410

readline() has to buffer the text, waiting for a new-line.

You'll always have a race condition - no amount of un-buffered streams will handle the fact that you're handling a line, then checking for exit, then reading a line, so if the subprocess exits while you're handling a line, you won't read anything else. In addition the shell is probably introducing its own buffering.

So you could either:

Use communicate() and give up the verbose output while the subprocess is running.
Make sure you continue reading after the process has exited, until you get EOF.

I would also suggest altering your code so that you don't have to use shell=True.

Unbuffered subprocess output (last line missing)

2 Answers2