subprocess.Popen.stdout - reading stdout in real-time (again)

Question

Again, the same question.
The reason is - I still can't make it work after reading the following:

• Real-time intercepting of stdout from another process in Python
• Intercepting stdout of a subprocess while it is running
• How do I get 'real-time' information back from a subprocess.Popen in python (2.5)
• catching stdout in realtime from subprocess

My case is that I have a console app written in C, lets take for example this code in a loop:

tmp = 0.0;   
printf("\ninput>>"); 
scanf_s("%f",&tmp); 
printf ("\ninput was: %f",tmp); 

It continuously reads some input and writes some output.

My python code to interact with it is the following:

p=subprocess.Popen([path],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
p.stdin.write('12345\n')
for line in p.stdout: 
    print(">>> " + str(line.rstrip())) 
    p.stdout.flush() 

So far whenever I read form p.stdout it always waits until the process is terminated and then outputs an empty string. I've tried lots of stuff - but still the same result.

I tried Python 2.6 and 3.1, but the version doesn't matter - I just need to make it work somewhere.

Answer 1

Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It's extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that's awaiting data before the system libraries flush it.

For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you'd probably get stalled processes (similar to what you're already seeing?)

You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you'll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the "EWOULDBLOCK" events. (I don't remember the exact Python exception that's raised for these).

A completely different approach would be for your parent to use the select module and os.fork() ... and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you'd be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.

Incidentally, .communicate, at least in Python's 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian's assertion to the contrary) but is limited to a much smaller value.

Answer 2

The bufsize=256 argument prevents 12345\n from being sent to the child process in a chunk smaller than 256 bytes, as it will be when omitting bufsize or inserting p.stdin.flush() after p.stdin.write(). Default behaviour is line-buffering.

In either case you should at least see one empty line before blocking as emitted by the first printf(\n...) in your example.

Answer 3

Push reading from the pipe into a separate thread that signals when a chunk of output is available:

How can I read all availably data from subprocess.Popen.stdout (non blocking)?

Answer 4

Your particular example doesn't require "real-time" interaction. The following works:

from subprocess import Popen, PIPE

p = Popen(["./a.out"], stdin=PIPE, stdout=PIPE)
output = p.communicate(b"12345")[0] # send input/read all output
print output,

where a.out is your example C program.

In general, for a dialog-based interaction with a subprocess you could use pexpect module (or its analogs on Windows):

import pexpect

child = pexpect.spawn("./a.out")
child.expect("input>>")
child.sendline("12345.67890") # send a number
child.expect(r"\d+\.\d+") # expect the number at the end
print float(child.after) # assert that we can parse it
child.close()

Answer 5

I had the same problem, and "proc.communicate()" does not solve it because it waits for process terminating.

So here is what is working for me, on Windows with Python 3.5.1 :

import subprocess as sp
myProcess = sp.Popen( cmd, creationflags=sp.CREATE_NEW_PROCESS_GROUP,stdout=sp.PIPE,stderr=sp.STDOUT)
while i<40:
        i+=1
        time.sleep(.5)
        out = myProcess.stdout.readline().decode("utf-8").rstrip()

I guess creationflags and other arguments are not mandatory (but I don't have time to test), so this would be the minimal syntax :

myProcess = sp.Popen( cmd, stdout=sp.PIPE)
for i in range(40)
            time.sleep(.5)
            out = myProcess.stdout.readline()