How to get subprocess' stdout data asynchronously?

Question

I wrote a simple python script for my application and predefined some fast commands like make etc.

I've written a function for running system commands (linux):

def runCommand(commandLine):
    print('############## Running command: ' + commandLine)
    p = subprocess.Popen(commandLine, shell = True, stdout = subprocess.PIPE)
    print (p.stdout.read().decode('utf-8'))

Everything works well except a few things:

I'm using cmake and it's output is colored. Any chances to save colors in output?
I can look at output after process has finished. For example, make runs for a long period of time but I can see the output only after full compilation. How to do it asynchronously?

score 12 · Accepted Answer · answered Jan 23 '12 at 19:58

I'm not sure about colors, but here's how to poll the subprocess's stdout one line at a time:

import subprocess
proc = subprocess.Popen('cmake', shell=True, stdout=subprocess.PIPE)
while proc.poll() is None:
    output = proc.stdout.readline()
    print output

Don't forget to read from stderr as well, as I'm sure cmake will emit information there.

Rob Wouters · Answer 2 · 2012-01-23T21:14:58.173

You're not getting color because cmake detects if its stdout is a terminal, if it's not it doesn't color its own output. Some programs give you an option to force coloring output. Unfortunately cmake does not, so you're out of luck there. Unless you want to patch cmake yourself.

Lots of programs do this, for example grep:

# grep test test.txt
test
 ^
 |
 |------- this word is red

Now pipe it to cat:

# grep test test.txt | cat
test
 ^
 |
 |------- no longer red

grep option --color=always to force color:

# grep test test.txt --color=always | cat
test
 ^
 |
 |------- red again

jcollado · Answer 3 · 2012-01-23T20:08:20.827

2

Regarding how to get the output of your process before it finishes, it should be possible to do that replacing:

p.stdout.read

with:

for line in p.stdout:

Regarding how to save colored output, there isn't anything special about that. For example, if the row output is saved to a file, then next time cat <logfile> is executed the console will interpret the escape sequences and displaye the colors as expected.

edited Jan 23 '12 at 20:08

answered Jan 23 '12 at 19:53

jcollado

39,419
8
102
133

with `readline()` I have a numbers in row instead of text, what's wrong? – Max Frai Jan 23 '12 at 19:57
1

@Ockonal Actually if you use the for loop, there's no need to use `readline`. You can find an example of both strategies [here](http://stackoverflow.com/q/2804543/183066). – jcollado Jan 23 '12 at 20:08
Actually `for line in p.stdout` is not good in Python 2 according to http://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line#comment11236752_2813530 – Piotr Dobrogost Oct 30 '12 at 19:40

score 1 · Answer 4 · answered Apr 22 '18 at 20:47

For CMake specifically, you can force color output using the option CLICOLOR_FORCE=1:

command = 'make CLICOLOR_FORCE=1'
args = shlex.split(command)
proc = subprocess.Popen(args, stdout=subprocess.PIPE)

Then print as in the accepted answer:

while proc.poll() is None:
    output = proc.stdout.readline()
    print(output.decode('utf-8'))

If you decode to utf-8 before printing, you should see colored output. If you print the result as a byte literal (i.e. without decoding), you should see the escape sequences for the various colors.

Consider trying the option universal_newlines=True:

proc = subprocess.Popen(args, stdout=subprocess.PIPE, universal_newlines=True)

This causes the call to proc.stdout.readline() to return a string instead of a byte literal so you can/must skip the call to decode().

score 0 · Answer 5 · answered Jan 23 '12 at 20:25

0

To do async output do something like: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Not sure if you can capture the coloured output. If you can get the escaped colour codes you may be able to.

answered Jan 23 '12 at 20:25

Nick

27,566
12
60
72

If we get the colour codes the terminal will translate them. The problem is though, most application distinguish between being attached to a terminal or being piped into another application. The only way to convince these applications to still produce the colour codes is by faking that the receiving app (in this case our python process) is in fact a tty and not a pipe. A task much easier said than done... – nickl- Feb 10 '13 at 16:43

score -1 · Answer 6 · edited May 23 '17 at 11:46

Worth noting here is the use of the script command as a pseudo terminal and be detected as a tty instead of redirect(pipe) file descriptor, see: bash command preserve color when piping

Works like a charm...

As per the example in the question simply let script execute cmake:

import subprocess
proc = subprocess.Popen('script cmake', shell=True, stdout=subprocess.PIPE)
while proc.poll() is None:
    output = proc.stdout.readline()
    print output

This tricks cmake into thinking it's being executed from a terminal and will produce the ANSI candy you're after.

nJoy!

How to get subprocess' stdout data asynchronously?

6 Answers6