Non-blocking read on subprocess PIPE in Python, one byte at a time

Question

I have implemented a variant on the code in this question:

A non-blocking read on a subprocess.PIPE in Python

To try and read the output in real time from this dummy program test.py:

import time, sys

print "Hello there"
for i in range(100):
    time.sleep(0.1)
    sys.stdout.write("\r%d"%i)
    sys.stdout.flush()
print
print "Go now or I shall taunt you once again!"

The variation on the other question is that the calling program must read character by character, not line by line, as the dummy program test.py outputs progress indication all on one line by use of \r. So here it is:

import sys,time
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # Python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    while True:
        buffersize = 1
        data = out.read(buffersize)
        if not data:
            break
        queue.put(data)
    out.close()

p = Popen(sys.executable + " test.py", stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # Thread dies with the program
t.start()

while True:
    p.poll()
    if p.returncode:
        break
    # Read line without blocking
    try:
        char = q.get_nowait()
        time.sleep(0.1)
    except Empty:
        pass
    else: # Got line
        sys.stdout.write(char)
        sys.stdout.flush()

print "left loop"
sys.exit(0)

Two problems with this

It never exits - p.returncode never returns a value and the loop is not left. How can I fix it?
It's really slow! Is there a way to make it more efficient without increasing buffersize?

According to the [docs for Popen](http://docs.python.org/2/library/subprocess.html#popen-constructor), setting `bufsize=1` means "line buffered". You probably want 0 (unbuffered). Also, why do you need the time.sleep(0.1) after `char = q.get_nowait()`? — Markku K., Jun 06 '13 at 15:38

score 2 · Accepted Answer · edited May 23 '17 at 11:47

2

As @Markku K. pointed out, you should use bufsize=0 to read one byte at a time.

Your code doesn't require a non-blocking read. You can simplify it:

import sys
from functools import partial
from subprocess import Popen, PIPE

p = Popen([sys.executable, "test.py"], stdout=PIPE, bufsize=0)
for b in iter(partial(p.stdout.read, 1), b""):
    print b # it should print as soon as `sys.stdout.flush()` is called
            # in the test.py
p.stdout.close()
p.wait()

Note: reading 1 byte at a time is very inefficient.

Also, in general, there could be a block-buffering issue that sometimes can be solved using pexpect, pty modules or unbuffer, stdbuf, script command-line utilities.

For Python processes you could use -u flag to force unbuffering (binary layer) of stdin, stdout, stderr streams.

edited May 23 '17 at 11:47

Community

1
1

answered Jun 06 '13 at 19:49

jfs

399,953
195
994
1,670

Cool, that clarifies things. Presumably if the above code is part of a GUI I do need non blocking reads or a separate thread to handle them? – Sideshow Bob Jun 07 '13 at 09:29
@SideshowBob: yes. If you want to get subprocess' output in a GUI in a portable manner; then [a thread+queue make sense](https://gist.github.com/zed/42324397516310c86288) or something like [`io_add_watch()`](https://gist.github.com/zed/8a255e81eb87431c0e63) if you know how much you can read without blocking e.g., if subprocess' output is line-buffered then `.readline()` inside `io_add_watch()` callback should *not* block. – jfs Jun 07 '13 at 11:45
`io_add_watch`-based solution is rather brittle if the fds are blocking – jfs Jun 07 '13 at 11:53
Have an accept :) I ended up using my original code with the modifications you suggest here - `bufsize=0`, `p.stdout.close()` and `p.wait()` – Sideshow Bob Jun 07 '13 at 14:08

Non-blocking read on subprocess PIPE in Python, one byte at a time

1 Answers1