Python 2 Subprocess: Cannot get output from readline

Question

I have the following C application

#include <stdio.h>

int main(void)
{
    printf("hello world\n");
    /* Go into an infinite loop here. */
    while(1);

    return 0;
}

And I have the following python code.

import subprocess
import time
import pprint


def run():
    command = ["./myapplication"]
    process = subprocess.Popen(command, stdout=subprocess.PIPE)
    try:
        while process.poll() is None:
            # HELP: This call blocks...
            for i in  process.stdout.readline():
                print(i)

    finally:
        if process.poll() is None:
            process.kill()


if __name__ == "__main__":
    run()

When I run the python code, the stdout.readline or even stdout.read blocks.

If I run the application using subprocess.call(program) then I can see "hello world" in stdout.

How can I read input from stdout with the example I have provided?

Note: I would not want to modify my C program. I have tried this on both Python 2.7.17 and Python 3.7.5 under Ubuntu 19.10 and I get the same behaviour. Adding bufsize=0 did not help me.

Probably a buffering issue: try adding a flush to the C application. — chepner, Jan 16 '20 at 18:12
It does seem to be a flushing issue, however, is there a way to avoid modifying the C program? Is there a way to run the subprocess as unbuffered? — Har, Jan 17 '20 at 10:36
I think `Popen(command, stdout=subprocess.PIPE, bufsize=0)` may do the trick. — chepner, Jan 17 '20 at 12:29
I coudlnt get it to work with bufsize=0 however I do understand the intention here... It should work... — Har, Feb 19 '20 at 17:58
I get the same behaviour on both python 2 and python 3 however I am using python 3.7.5. and python 2.7.17 I would like it to work on both 2 and 3. — Har, Feb 19 '20 at 20:53
The buffer exists on the side of the C process as well. I don't think you can force other process to flush its buffers, unless you want to get creative (and non-portable and unmaintainable). — BartoszKP, Feb 19 '20 at 21:01
However without the stdout=… the C program does flush its buffer... or at least I think it is the C program and not something else as I can see "hello world" being printed. It seems that adding in the stdout=… changes the behaviour of flushing.. — Har, Feb 19 '20 at 21:08
Cross-site dupe: [Turn off buffering in pipe](https://unix.stackexchange.com/q/25372/12321). On-site dupe but look to the second answer, not the accepted answer: [Read streaming input from subprocess.communicate()](https://stackoverflow.com/a/17698359/674039) — wim, Feb 19 '20 at 23:56

Timo Hartmann · Answer 1 · 2020-02-19T21:27:43.673

The easiest way is to flush buffers in the C program

...
printf("hello world\n");
fflush(stdout);
while(1);
...

If you don't want to change the C program, you can manipulate the libc buffering behavior from outside. This can be done by using stdbuf to call your program (linux). The syntax is "stdbuf -o0 yourapplication" for zero buffering and "stdbuf -oL yourapplication" for line buffering. Therefore in your python code use

...
command = ["/usr/bin/stdbuf","-oL","pathtomyapplication"]
process = subprocess.Popen(command, stdout=subprocess.PIPE)
...

or

...
command = ["/usr/bin/stdbuf","-o0","pathtomyapplication"]
process = subprocess.Popen(command, stdout=subprocess.PIPE)
...

wim · Accepted Answer · 2020-02-20T02:34:53.943

Applications built using the C Standard IO Library (built with #include <stdio.h>) buffer input and output (see here for why). The stdio library, like isatty, can tell that it is writing to a pipe not a TTY and so it chooses block buffering instead of line buffering. Data is flushed when the buffer is full, but "hello world\n" is not filling the buffer so it's not flushed.

One way around is shown in Timo Hartmann answer, using stdbuf utility. This uses an LD_PRELOAD trick to swap in its own libstdbuf.so. In many cases that is a fine solution, but LD_PRELOAD is kind of a hack and does not work in some cases, so it may not be a general solution.

Maybe you want to do this directly in Python, and there are stdlib options to help here, you can use a pseudo-tty (docs py2, docs py3) connected to stdout instead of a pipe. The program myapplication should enable line buffering, meaning that any newline character flushes the buffer.

from __future__ import print_function
from subprocess import Popen, PIPE
import errno
import os
import pty
import sys

mfd, sfd = pty.openpty()
proc = Popen(["/tmp/myapplication"], stdout=sfd)
os.close(sfd)  # don't wait for input
while True:
    try:
        output = os.read(mfd, 1000)
    except OSError as e:
        if e.errno != errno.EIO:
            raise
    else:
        print(output)

Note that we are reading bytes from the output now, so we can not necessarily decode them right away!

See Processing the output of a subprocess with Python in realtime for a blog post cleaning up this idea. There are also existing third-party libraries to do this stuff, see ptyprocess.

Python 2 Subprocess: Cannot get output from readline

2 Answers2