Reliable non blocking reads from subprocess stdout

Question

Note: I have a process that writes one line to stdout ("print("hello")) and waits for raw_input. I run this process with p = subprocess.Popen, then call p.stdout.readline()....this blocks indefinitely. I am setting shell=False...Why can I not read that first output?

p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin = subprocess.PIPE)
    print(p.stdout.readline())

I have seen this thread about non blocking io for subprocess. I am trying to create an interactive subprocess object that reads and writes to the process. I found that stdout is blocking, even though there is output sent to stdout. To test this, I wrote a program that spits increasing integers to stdout and called this via subprocess. I then used the non blocking method involving a Queue to read stdout. What I found was that, despite the fact that the process is spitting 1000 lines to stdout per second, Stdout blocks for long periods at a time, returning a line completely randomly...one in every thousand, then one in 10 etc. Any thoughts on why this might be happening? If I slow down the printer to print once every 0.5 seconds, I never get a read from stdout. Its like it needs 1000 writes to sdtout before it responds with a single line.

Note: I put a sleep in so there was time between starting the listen thread and reading from it. Now, I always return from stdout, but I get a random single line. I guess I don't understand stdout. Does it not buffer? I mean, I want to get everything that is spit out by my process, but it seems like it only saves the last line. I can't find documentation on these PIPE things.

  def getStdOut(self):
        '''
            reads standard out
        '''
        return self.nonBlockingRead()


    def enqueue_output(self, out, queue, kill):
        line = ""
        kill = True
        while kill:
            line = out.readline()
            queue.put(line)

#             for line in iter(out.readline, b''):
#                 queue.put(line)
#             out.close()

    def nonBlockingRead(self):
        '''
             taken from the internet
        '''
        import sys
        from subprocess import PIPE, Popen
        from threading  import Thread
        sleep(0.5) #I inserted this later
        try:
            from Queue import Queue, Empty
        except ImportError:
            from queue import Queue, Empty  # python 3.x

        ON_POSIX = 'posix' in sys.builtin_module_names

        killThread = False                
        self.q = Queue()
        t = Thread(target=self.enqueue_output, args=(self.theProcess.stdout, self.q, killThread))
        t.daemon = True # thread dies with the program
        t.start()

        line = ''
        try:  
            line = self.q.get_nowait() # or q.get(timeout=.1)
        except Exception as e:
            print(e)
            print('no output yet')
        killThread = True
        return line




if __name__ == "__main__" :
    '''
        unit tests
    '''
    import time
    cmd = 'python "C:\InteractiveTest.py"'
    aSubProc = subProcessWStdIn(cmd, 10)
    while True :    
#        print(aSubProc.writeToProcess('y'))
        print(aSubProc.getStdOut())

possible duplicate of [read subprocess stdout line by line](http://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line) — tutuDajuju, Feb 26 '14 at 18:21
I have moved on to trying winpexpect, but amazingly, this blocks too! If all your proces does is print one line then waits for an input, winpexpect blocks and then throws an exception!!! — user442920, Feb 26 '14 at 20:13

tutuDajuju · Accepted Answer · 2014-02-26T20:28:06.917

readline reads a single line from the file-like object PIPE, to read it all of it, simply wrap it in a while loop. You should also call sleep after each read to save on CPU cycles.

Here is a simple example:

import subprocess

p = subprocess.Popen(
    ['ls', '-lat'],
    shell=False,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    stdin=subprocess.PIPE
)
while True:
    line = p.stdout.readline()
    if line == '':
        break
    print(line.strip())  # remove extra ws between lines

EDIT:

woah, sorry, I completely missed the part you were trying to read input in that other process...

So, fn your other process, looks something like:

print('Hello')
in = raw_input()

Then the print actually sends the content to the file-like object PIPE you passed earlier which has it's own buffering mechanism. This behavior is explained in the print() function docs

To solve this simply add a sys.stdout.flush() between your print and raw_input:

print('Hello')
sys.stdout.flush()  # "flush" the output to our PIPE
in = raw_input()

readline() is blocking, so depending on the application. Comparing the line with a empty line ('') will never be occur. Meaning the while loop will keep running (never ends) — Melroy van den Berg, Nov 26 '14 at 12:55
I agree, the condition is completely arbitrary and it's your responsibility to break out of the reading loop when your application requires it — tutuDajuju, Nov 26 '14 at 13:47

Reliable non blocking reads from subprocess stdout

1 Answers1

EDIT:

Linked