subprocess not returning until command complete

Question

I am trying to get a real time output from a bash command so i can play with the data easier.

In this code, the command iostat 1 works fine and prints output at 1 second. The command sar -u 1 20, which runs as expected on the command line (printing 1 line every second up to 20), waits until the command is complete ~20 seconds, before printing each line with a .1 second delay.

I am planning on running these commands indefinitely, and need this part to work. Any thoughts on whats wrong? I am on OSX.

import subprocess
import time

# run a command and constantly check for new output, if you find it, print it and continue
# if the command is done, break out
try:
    command="sar -u 1 20"
    process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
    while True:
        time.sleep(.1) # so i don't kill the cpu on the machine i'm checking
        output = process.stdout.readline()
        #if process.poll() is not None:
        #    break
        if output:
            print output.strip()
except KeyboardInterrupt:
    print 'Exiting...'
return_code = process.poll()
print return_code

score 1 · Accepted Answer · answered Nov 20 '15 at 16:22

1

sar detects that its standard output is not a terminal and buffers its output. It doesn't produce much output, so the buffer does not fill up enough to be flushed to the pipe before it times out.

If you install GNU coreutils, you can use the stdbuf command to disable buffering for standard output. (If you install it via Homebrew, it is installed as gstdbuf.)

command = "stdbuf -o 0 sar -u 1 20"

I'm not sure if there is a comparable solution using tools included with Mac OS X.

answered Nov 20 '15 at 16:22

chepner

497,756
71
530
681

This looks like what my problem was, i cant seem to install it, but it should be installed via coreutils. I found http://apple.stackexchange.com/questions/193141/to-use-stdbuf-from-homebrews-coreutils which describes this tool. – user1601716 Nov 20 '15 at 16:37
After `brew install coreutils`, you need to use `gstdbuf`; the package prefixes all the command names with `g` to avoid any conflict with the standard tools (`gls` instead of `ls`, etc. `stdbuf` wouldn't conflict, but apparently the packager wanted to be consistent.) – chepner Nov 20 '15 at 16:46
having issues installing coreutils, looks like a bug with 'make install', there is a file with an infinite name :(, but this looks to have been the problem, I can get around this for now. Thanks! – user1601716 Nov 20 '15 at 16:50

score 0 · Answer 2 · edited May 23 '17 at 11:48

0

from: https://stackoverflow.com/a/17698359/16148

for Python 2:

from subprocess import Popen, PIPE

p = Popen(["cmd", "arg1"], stdout=PIPE, bufsize=1)
with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print line,
p.wait() # wait for the subprocess to exit

edited May 23 '17 at 11:48

Community

1
1

answered Nov 20 '15 at 16:07

Corey Goldberg

59,062
28
129
143

Not working, code is cleaner, but the sar command still waits until its done before it starts to print. – user1601716 Nov 20 '15 at 16:14

subprocess not returning until command complete

2 Answers2