subprocess.Popen() stdout and stderr handling

Question

How can I handle a process stderr?

proc = subprocess.Popen('ll'.split(), stdout=subprocess.PIPE)

for i in proc.stdout:
   print(i)

Right now I am streaming the output but I am not sure how to properly deal with a potential error that might occur.

I wanted to use out, err = proc.communicate() but my out could be a very very very long string

How long a string are we talking? Up to several megabytes, buffering in memory is going to be easier and more performant, unless you need to consume the results as they're being generated. — Daniel Pryden, Oct 23 '18 at 00:45
@DanielPryden potentially "{owner} {permissions} {path}/{name}" multiplied by the number of hours in 4 years. I am also processing each string because I just need a substring of the string separated by `\n` — Liondancer, Oct 23 '18 at 00:47
https://stackoverflow.com/questions/2715847/python-read-streaming-input-from-subprocess-communicate proc.stdout.readline() — Serge, Oct 23 '18 at 00:47
If you're actually getting filesystem information, don't use a subprocess for that! Use `os.listdir()` (or maybe `os.walk()` if you need subdirectories) plus `os.stat()` to collect the extra information. Not only will it save you messing about with a subprocess and parsing strings, it will also likely be faster overall. — Daniel Pryden, Oct 23 '18 at 00:50
@DanielPryden No it isn't getting filesystem information. `ll` was just a nice example. I am actually running an `HDFS -ls` command but `ll` was a good substitute that I think more people would understand and had similar output — Liondancer, Oct 23 '18 at 00:51
Possible duplicate of [Run command and get its stdout, stderr separately in near real time like in a terminal](https://stackoverflow.com/questions/31926470/run-command-and-get-its-stdout-stderr-separately-in-near-real-time-like-in-a-te) — Daniel Pryden, Oct 23 '18 at 01:04

score 1 · Answer 1 · answered Oct 23 '18 at 01:46

If you know what error messages to expect, then one answer is to pass subprocess.STDOUT to the stderr argument of Popen, so that your stderr messages are in the stdout stream:

proc = subprocess.Popen('ll'.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

for i in proc.stdout:
   print(i)
   # check for error message strings and do something with them

Or if you don't care about the stdout messages then just iterate over stderr instead:

dnull = open(os.devnull, 'w')
proc = subprocess.Popen('ll'.split(), stdout=dnull, stderr=subprocess.PIPE)

for i in proc.stderr:
   print(i)
   # check for error message strings and do something with them

subprocess.Popen() stdout and stderr handling

1 Answers1