Running shell command in python and reading output

Question

I have the following:

cmd = "ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'".split(' ')
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out

When I run the command in the console (outside of python), I get the desired output. Running this above code in python prints a blank line. I am assuming there is something up with the cmd (specifically the | operator) but I can't be sure.

I need to achieve this with the standard Python 2.6.6 install (no additional modules)

possible duplicate of [Python subprocess command with pipe](https://stackoverflow.com/questions/13332268/python-subprocess-command-with-pipe) — RomanHotsiy, Jul 16 '14 at 11:24
If your operating system provides a pgrep command, using that is much better practice than filtering output of ps in this way. — Charles Duffy, Jul 16 '14 at 11:24
possible duplicate of [Running shell command from python and capturing the output](http://stackoverflow.com/questions/4760215/running-shell-command-from-python-and-capturing-the-output) — Charles Duffy, Jul 16 '14 at 11:26
VERY clearly not a duplicate - the specific problem was around the pipe operator. — Cheetah, Jul 16 '14 at 12:50

score 3 · Accepted Answer · answered Jul 16 '14 at 10:41

You need to use a single call to Popen() for each piece of the original command, as connected by the pipe, as in

import subprocess

p1 = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE,  stderr=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "java -jar"], stdin=p1.stdout, stdout=subprocess.PIPE,  stderr=subprocess.PIPE)
p3 = subprocess.Popen(["grep", "-v", "grep"], stdin=p2.stdout, stdout=subprocess.PIPE,  stderr=subprocess.PIPE)
p4 = subprocess.Popen(["awk", "//{print $2}"], stdin=p3.stdout, stdout=subprocess.PIPE,  stderr=subprocess.PIPE)
out, err = p4.communicate()

print out

The subprocess documentation has an in-depth discussion.

score 3 · Answer 2 · answered Jul 16 '14 at 11:18

Popen by default only executes executables, not shell command lines. When you pass the list of arguments to Popen they should call one executable with its arguments:

import subprocess

proc = subprocess.Popen(['ps', 'aux'])

Also note that you should not use str.split to split a command, because:

>>> "ps aux | grep 'java -jar'     | grep -v grep | awk '//{print $2}'".split(' ')
['ps', 'aux', '|', 'grep', "'java", "-jar'", '', '', '', '', '|', 'grep', '-v', 'grep', '|', 'awk', "'//{print", "$2}'"]

Note how:

The arguments that were quoted (e.g. 'java -jar') are splitted.
If there is more than one consecutive space you get some empty arguments.

Python already provides a module that knows how to split a command line in a reasonable manner, it's shlex:

>>> shlex.split("ps aux | grep 'java -jar'     | grep -v grep | awk '//{print $2}'")
['ps', 'aux', '|', 'grep', 'java -jar', '|', 'grep', '-v', 'grep', '|', 'awk', '//{print $2}']

Note how quoted arguments were preserved, and multiple spaces are handled gracefully. Still you cannot pass the result to Popen, because Popen will not interpret the | as a pipe by default.

If you want to run a shell command line (i.e. use any shell feature such as pipes, path expansion, redirection etc.) you must pass shell=True. In this case you should not pass a list of strings as argumento to Popen, but only a string that is the complete command line:

proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)

If you pass a list of strings with shell=True its meaning is different: the first element should be the complete command line, while the other elements are passed as options to the shell used. For example on my machine the default shell (sh) has an -x option that will display on stderr all the processes that gets executed:

>>> from subprocess import Popen
>>> proc = Popen(['ps aux | grep python3', '-x'], shell=True)
>>> 
username   7301  0.1  0.1  39440  7408 pts/9    S+   12:57   0:00 python3
username   7302  0.0  0.0   4444   640 pts/9    S+   12:58   0:00 /bin/sh -c ps aux | grep python3 -x
username   7304  0.0  0.0  15968   904 pts/9    S+   12:58   0:00 grep python3

Here you can see that a /bin/sh was started that executed the command ps aux | python3 and with an option of -x.

(This is all documented in the documentation for Popen).

This said, one way to achieve what you want is to use subprocess.check_output:

subprocess.check_output("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True)

However this isn't available in python<2.7 so you have to use Popen and communicate():

proc = subprocess.Popen("ps aux | grep 'java -jar' | grep -v grep | awk '//{print $2}'", shell=True, stdout=subprocess.PIPE)
out, err = proc.communicate()

The alternative is to avoid using shell=True (which is generally a very good thing, since shell=True introduces some security risks) and manually write the pipe using multiple processes:

from subprocess import Popen, PIPE

ps = Popen(['ps', 'aux'], stdout=PIPE)
grep_java = Popen(['grep', 'java -jar'], stdin=ps.stdout, stdout=PIPE)
grep_grep = Popen(['grep', '-v', 'grep'], stdin=grep_java.stdout, stdout=PIPE)
awk = Popen(['awk', '//{print $2}'], stdin=grep_grep.stdout, stdout=PIPE)
out, err = awk.communicate()

grep_grep.wait()
grep_java.wait()
ps.wait()

Note that if you don't care for the standard error you can avoid specifying it. It will then inherit the one of the current process.

score 1 · Answer 3 · answered Jul 16 '14 at 10:52

There is just one command in the shell pipeline which can't be easily replaced by Python code. So you can start several external processes and connect their inputs and outputs, but I would just start the ps aux and add some Python code to filter and extract the desired data:

from subprocess import PIPE, Popen


def main():
    process = Popen(['ps', 'aux'], stdout=PIPE)
    pids = [
        line.split(None, 2)[1] for line in process.stdout if 'java -jar' in line
    ]
    process.wait()
    print '\n'.join(pids)


if __name__ == '__main__':
    main()

Running shell command in python and reading output

3 Answers3