2

I'm trying to print stdout in realtime for a subprocess but it looks like stdout is buffered even with bufsize=0 and I can't figure out how to make it work, I always have a delay.

The code I tried :

p = subprocess.Popen(cmd, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=0)
line = p.stdout.readline()
while line:
    sys.stdout.write(line)
    sys.stdout.flush()
    # DO OTHER STUFF
    line = p.stdout.readline()

Also tried with for line in iter(p.stdout.readline, b'') instead of the while loop and with read(1) instead of readline(). Always the same result, the output gets delayed by a lot of seconds or minutes and multiple lines appear suddenly at once.

What I think happens :

bufsize is set to 0 ( it is set to 0 by default according to the docs ) so the lines piped top.stdout should be available immediately. But since p.stdout.readline() doesn't return immediately when a new line is piped, that means that it IS buffered, hence the multiple lines at once when the buffer is finally flushed to p.stdout.

What can I do to make it work ?

Nolhian
  • 574
  • 2
  • 7
  • 18
  • related: [catching stdout in realtime from subprocess](http://stackoverflow.com/q/18273962/4279) – jfs Nov 06 '13 at 20:00
  • 1
    related: [Python subprocess readlines() hangs](http://stackoverflow.com/q/12419198/4279) – jfs Nov 06 '13 at 20:25
  • Sorry, saw your latest comment after finding the solution. Still upvoted because the answer matches my problem. – Nolhian Nov 07 '13 at 14:35

3 Answers3

2

Thanks to pobrelkey who found the source of the problem. Indeed, the delay is due to the fact that the child is buffering its write to stdout because it is not writing to a tty. The child uses stdio which is line buffered when writing to a tty, else it is fully buffered.

I managed to get it to work by using pexpect instead of subprocess. pexpect uses a pseudo-tty and that's exactly what we need here :

p = pexpect.spawn(cmd,args,timeout=None) 
line = p.readline() 
while line:
    sys.stdout.write(line)
    sys.stdout.flush()
    # DO OTHER STUFF
    line = p.readline()

Or even better in my case :

p = pexpect.spawn(cmd,args,timeout=None,logfile=sys.stdout)
line = p.readline() 
while line:
    # DO OTHER STUFF
    line = p.readline()

No more delay !

More infos about pexpect : wiki

Nolhian
  • 574
  • 2
  • 7
  • 18
  • 1
    if the child uses stdio then you could make it to switch to line-buffering using [`stdbuf -oL` (or `unbuffer`, `script`)](http://unix.stackexchange.com/q/25372/1321). There is a [code example in my answer](http://stackoverflow.com/a/12471855/4279). See also [Python: read streaming input from subprocess.communicate()](http://stackoverflow.com/q/2715847/4279) – jfs Nov 07 '13 at 15:05
  • Indeed, your post is awesome, I really wish I'd found it before writing this thread. I looked up a lot of "getting output in realtime with subprocess" threads but missed this one. Is there a preferred method between `pexpect`, `stdbuf` and `tty` ? – Nolhian Nov 07 '13 at 20:37
  • 1
    it depends. `stdbuf` variant requires that the external program (`stdbuf`) is present. `pexpect` is a 3rd party pure Python module so you have to learn its API. `pty` is in stdlib but it might be harder to use (judging by the [code example](http://stackoverflow.com/a/12471855/4279)). None works on Windows as is. – jfs Nov 08 '13 at 05:48
0

I would first make sure the subprocess itself doesn't buffer its output. If the subprocess is in turn a Python program, proceed to the paragraph below to see how to disable output buffering for Python processes.

As per Python, usually the problem is that Python by default buffers stderr and stdout even if you explicitly .flush() it from the code. The solution is to pass -u to Python when starting your program.

Also, you can just do for line in p.stdout instead of the tricky while loop.

P.S. actually I tried running your code (with cmd = ['cat', '/dev/urandom']) and without -u and it outputted everything in real time already; this is on OS X 10.8.

Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111
  • Just tried with -u and still the same. I guess that cat /dev/urandom would work because it generates a lot of data so the buffer flushes eventually in p.stdout. I'll try to see if I can dig more infos. – Nolhian Nov 06 '13 at 17:59
  • I tried with something slower such as e.g. `/var/log/system.log` and I'm still getting the output smoothly. Which platform are you on btw? – Erik Kaplun Nov 06 '13 at 18:00
  • could you provide a code example that demonstrates the `.flush()` behaviour? – jfs Nov 06 '13 at 20:07
  • @Erik Allik : Debian 7 64 bits – Nolhian Nov 07 '13 at 13:18
0

If you just want stdout of your child process to go to your stdout, why not just have the child process inherit stdout from your process?

subprocess.Popen(cmd, stdout=None, stderr=subprocess.STDOUT)
pobrelkey
  • 5,853
  • 20
  • 29
  • Indeed but I actually do other stuff other than outputting to stdout. Sorry I edited my post. – Nolhian Nov 06 '13 at 17:40
  • 2
    Ah. Then it will depend on whether the child process buffers its writes to stdout - often this will automatically happen if a process isn't writing to a tty. In which case you'll need to modify the child process instead of your script, or figure out some way of piping its output through a pty (expect or the pty module might help here). Sorry. – pobrelkey Nov 06 '13 at 17:48