invoking pipe command with Popen in Python

Question

What is the correct way to invoke a command containing multiple pipes to Popen so that its output can be read? I tried:

Popen(shlex.split("mycmd arg1 | mysecondcmd - | thirdcmd -", stdout=PIPE)")

But I don't believe shlex.split is right here. What's the correct syntax?

For one thing, you're passing `stdout=PIPE` as an argument to `shlex.split`, not to the `Popen` constructor. For another, you've got an odd number of `"` characters; presumably it's the last one that's extraneous. Also, if you want to use shell features, you have to pass `shell=True`. (You usually don't want to spawn a shell to pipe the commands together, but if you want to, you can.) — abarnert, Jan 28 '13 at 20:06
Possible duplicate of [link several Popen commands with pipes](http://stackoverflow.com/questions/7389662/link-several-popen-commands-with-pipes) — tripleee, Aug 21 '16 at 13:51

score 6 · Accepted Answer · answered Jan 28 '13 at 18:31

6

You have a few options -- You can pass shell=True:

Popen('command1 | command2 | command3',shell=True)

Or, you can break it up into a bunch of Popen calls hooking their stdout to the next Popen's stdin as demonstrated in the documentation.

answered Jan 28 '13 at 18:31

mgilson

300,191
65
633
696

unutbu · Answer 2 · 2013-01-28T18:51:53.540

Using the sh module, pipes become function composition:

import sh
output = sh.thirdcmd(sh.mysecondcmd(sh.mycmd("arg1")))

If you want to do it with subprocess without shell = True, there is an example in the docs which shows how to write shell pipelines using subprocess.Popen. Note that you are supposed to close the proc.stdouts so that SIGPIPEs can be received properly:

import subprocess
proc1 = subprocess.Popen(shlex.split('mycmd arg1'), stdout = subprocess.PIPE)
proc2 = subprocess.Popen(shlex.split('mysecondcmd'), stdin = proc1.PIPE,
                         stdout = subprocess.PIPE)
proc3 = subprocess.Popen(shlex.split('thirdcmd'), stdin = proc2.PIPE,
                         stdout = subprocess.PIPE)

# Allow proc1 to receive a SIGPIPE if proc2 exits.
proc1.stdout.close()
# Allow proc2 to receive a SIGPIPE if proc3 exits.
proc2.stdout.close()
out, err = proc3.communicate()

This might look like a lot more work than using shell = True. The reason why you might want to avoid shell = True is because it can be a security risk (page down to the "Warning" box), especially if you are running a command supplied by a (potentially malicious) user.

invoking pipe command with Popen in Python

2 Answers2