1

I have a requirement to run a shell command in Python, and simply print out its output back to the screen, without changing what is written to stdout and stderr, in real time.

I can see that similar questions have been asked before but I can't find where this was ever answered simply.

I have tried things like

import subprocess, sys
  
command = "some long shell command".split(" ")

proc = subprocess.Popen(
    command,
    stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE
)

while True:
    out = proc.stdout.read(1)

    if out:
        sys.stdout.write(str(out))
        sys.stdout.flush()

    err = proc.stderr.read(1)

    if err:
        sys.stderr.write(str(err))
        sys.stderr.flush()

    if proc.poll():
        break

This does not work, but it probably helps to understand what I am trying to do.

Alex Harvey
  • 14,494
  • 5
  • 61
  • 97
  • In short: _Run a thread for each_. The code here doesn't work because `proc.stdout.read(1)` is a blocking call -- if there's nothing to read it'll wait until there is -- unless you explicitly configure the file descriptor to be asynchronous. – Charles Duffy Jun 18 '21 at 17:12
  • ...I'd be really astonished if none of the linked duplicates discuss threading; it's the easiest, most obvious approach. If you have your two threads put items like `('E', ...string)` or `('O', ...string...)` into a queue, you can then have a loop that reads off that queue in the main thread if that's what makes you happy. – Charles Duffy Jun 18 '21 at 17:13
  • ...the alternative to threading is using `select()` (or one of its newer equivalents) to tell you if a read will block before attempting it, and I see that one of the duplicates does that too. – Charles Duffy Jun 18 '21 at 17:15
  • Indeed, the duplicate you linked https://stackoverflow.com/questions/21617249/read-subprocess-stdout-and-stderr-concurrently _does_ teach the threading approach; and the OP at https://stackoverflow.com/questions/27945934/read-stdout-and-stderr-from-subprocess-line-by-line-simulanously-without-one-sel is trying to use the `select()` approach in their question, which is at least a good starting point. – Charles Duffy Jun 18 '21 at 17:17
  • Can you show how you tried to apply the threading approach taught by the answer you linked, and a specific problem you encountered when attempting to do so? – Charles Duffy Jun 18 '21 at 17:18
  • 1
    (BTW, `"some long shell command".split()` is inherently buggy -- you don't want `'foo --arg "bar baz"'` to become `["foo", "--arg", '"bar', 'baz"']`, but that's exactly what it does; `shlex.split()` is at least a bit less bad, though it's by far best to just write an explicit argument list by hand instead of trying to use strings at all in this context). – Charles Duffy Jun 18 '21 at 17:20
  • 1
    https://stackoverflow.com/questions/31833897/python-read-from-subprocess-stdout-and-stderr-separately-while-preserving-order is another duplicate, and arguably the best I've seen yet for teaching the `select()`-type approach. The answer https://stackoverflow.com/a/61585093/14122 is so on-point I think it makes the question fair to close. – Charles Duffy Jun 18 '21 at 17:21
  • @CharlesDuffy Thanks for finding the right answer, amongst hundreds. It is hard for someone like me to know that an answer upvoted only 3 times by a user of 300 rep is the best answer there is! – Alex Harvey Jun 19 '21 at 01:50

0 Answers0