5

I want to capture stdout from a long-ish running process started via subprocess.Popen(...) so I'm using stdout=PIPE as an arg.

However, because it's a long running process I also want to send the output to the console (as if I hadn't piped it) to give the user of the script an idea that it's still working.

Is this at all possible?

Cheers.

Steve Folly
  • 8,327
  • 9
  • 52
  • 63
  • Duplicate: http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess and http://stackoverflow.com/questions/874815/how-do-i-get-real-time-information-back-from-a-subprocess-popen-in-python-2-5 – S.Lott Aug 16 '09 at 01:10
  • related: [Python: read streaming input from subprocess.communicate()](http://stackoverflow.com/a/17698359/4279) – jfs Oct 16 '14 at 20:02

5 Answers5

5

The buffering your long-running sub-process is probably performing will make your console output jerky and very bad UX. I suggest you consider instead using pexpect (or, on Windows, wexpect) to defeat such buffering and get smooth, regular output from the sub-process. For example (on just about any unix-y system, after installing pexpect):

>>> import pexpect
>>> child = pexpect.spawn('/bin/bash -c "echo ba; sleep 1; echo bu"', logfile=sys.stdout); x=child.expect(pexpect.EOF); child.close()
ba
bu
>>> child.before
'ba\r\nbu\r\n'

The ba and bu will come with the proper timing (about a second between them). Note the output is not subject to normal terminal processing, so the carriage returns are left in there -- you'll need to post-process the string yourself (just a simple .replace!-) if you need \n as end-of-line markers (the lack of processing is important just in case the sub-process is writing binary data to its stdout -- this ensures all the data's left intact!-).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
2

S. Lott's comment points to Getting realtime output using subprocess and Real-time intercepting of stdout from another process in Python

I'm curious that Alex's answer here is different from his answer 1085071. My simple little experiments with the answers in the two other referenced questions has given good results...

I went and looked at wexpect as per Alex's answer above, but I have to say reading the comments in the code I was not left a very good feeling about using it.

I guess the meta-question here is when will pexpect/wexpect be one of the Included Batteries?

Community
  • 1
  • 1
user135331
  • 229
  • 1
  • 5
1

Can you simply print it as you read it from the pipe?

RichieHindle
  • 272,464
  • 47
  • 358
  • 399
  • Ah! (light bulb moment!) - p.stdout is a file like object. So I just need to call readline() on it in a loop? Right? What's the best way to exit the loop - when p.stdout returns empty string? Should I still do p.wait() after just to be sure? – Steve Folly Aug 16 '09 at 07:44
  • 1
    @Steve: An empty string returned from reading p.stdout means that the child has closed its stdout, but not necessarily that it's terminated. p.wait() will wait for it to terminate. – RichieHindle Aug 16 '09 at 21:12
1

Alternatively, you can pipe your process into tee and capture only one of the streams. Something along the lines of sh -c 'process interesting stuff' | tee /dev/stderr.

Of course, this only works on Unix-like systems.

Nathan
  • 4,009
  • 2
  • 20
  • 21
AI0867
  • 376
  • 4
  • 11
1

Inspired by pty.openpty() suggestion somewhere above, tested on python2.6, linux. Publishing since it took a while to make this working properly, w/o buffering...

def call_and_peek_output(cmd, shell=False):
    import pty, subprocess
    master, slave = pty.openpty()
    p = subprocess.Popen(cmd, shell=shell, stdin=None, stdout=slave, close_fds=True)
    os.close(slave)
    line = ""
    while True:
        try:
            ch = os.read(master, 1)
        except OSError:
            # We get this exception when the spawn process closes all references to the
            # pty descriptor which we passed him to use for stdout
            # (typically when it and its childs exit)
            break
        line += ch
        sys.stdout.write(ch)
        if ch == '\n':
            yield line
            line = ""
    if line:
        yield line

    ret = p.wait()
    if ret:
        raise subprocess.CalledProcessError(ret, cmd)

for l in call_and_peek_output("ls /", shell=True):
    pass
maximk
  • 188
  • 1
  • 4