Pipe subprocess standard output to a variable

Question

I want to run a command in pythong, using the subprocess module, and store the output in a variable. However, I do not want the command's output to be printed to the terminal. For this code:

def storels():
   a = subprocess.Popen("ls",shell=True)
storels()

I get the directory listing in the terminal, instead of having it stored in a. I've also tried:

 def storels():
       subprocess.Popen("ls > tmp",shell=True)
       a = open("./tmp")
       [Rest of Code]
 storels()

This also prints the output of ls to my terminal. I've even tried this command with the somewhat dated os.system method, since running ls > tmp in the terminal doesn't print ls to the terminal at all, but stores it in tmp. However, the same thing happens.

Edit:

I get the following error after following marcog's advice, but only when running a more complex command. cdrecord --help. Python spits this out:

Traceback (most recent call last):
  File "./install.py", line 52, in <module>
    burntrack2("hi")
  File "./install.py", line 46, in burntrack2
    a = subprocess.Popen("cdrecord --help",stdout = subprocess.PIPE)
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Just a note, using `shell=true` is discouraged in the Python docs. http://docs.python.org/2/library/subprocess.html#frequently-used-arguments — SamHuckaby, Dec 09 '13 at 16:13
Use this link `https://stackoverflow.com/a/75175057/12780274` — henrry, Jan 19 '23 at 16:04

score 160 · Accepted Answer · edited May 23 '17 at 12:34

160

To get the output of ls, use stdout=subprocess.PIPE.

>>> proc = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> output = proc.stdout.read()
>>> print output
bar
baz
foo

The command cdrecord --help outputs to stderr, so you need to pipe that indstead. You should also break up the command into a list of tokens as I've done below, or the alternative is to pass the shell=True argument but this fires up a fully-blown shell which can be dangerous if you don't control the contents of the command string.

>>> proc = subprocess.Popen(['cdrecord', '--help'], stderr=subprocess.PIPE)
>>> output = proc.stderr.read()
>>> print output
Usage: wodim [options] track1...trackn
Options:
    -version    print version information and exit
    dev=target  SCSI target to use as CD/DVD-Recorder
    gracetime=# set the grace time before starting to write to #.
...

If you have a command that outputs to both stdout and stderr and you want to merge them, you can do that by piping stderr to stdout and then catching stdout.

subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

As mentioned by Chris Morgan, you should be using proc.communicate() instead of proc.read().

>>> proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> out, err = proc.communicate()
>>> print 'stdout:', out
stdout: 
>>> print 'stderr:', err
stderr:Usage: wodim [options] track1...trackn
Options:
    -version    print version information and exit
    dev=target  SCSI target to use as CD/DVD-Recorder
    gracetime=# set the grace time before starting to write to #.
...

edited May 23 '17 at 12:34

Community

1
1

answered Dec 22 '10 at 23:49

moinudin

134,091
45
190
216

Alright, that works for 'ls', but later on I ran into a different problem, when trying to run a different command. When I try to run "cdrecord --help" with that method, I get a traceback error. – Insomaniacal Dec 23 '10 at 00:04
Thanks a ton!! That did it. If you don't mind me asking, how would you differentiate between using the stdout method and this stderr method? – Insomaniacal Dec 23 '10 at 00:15
1

@Insomaniacal In this case, I knew usage is generally dumped to stderr. But in general, a simple test is to `cmd > /dev/null` if you still see output it's going to stderr. Confirm by piping stderr `cmd 2> /dev/null` and it should vanish into `/dev/null`. – moinudin Dec 23 '10 at 00:18
1

@Insomaniacal: in general `stdout` will be used, sometimes but sometimes `stderr`; in general, use `stdout` unless it doesn't work and then try `stderr`. I believe it can be arranged in some terminals to colour stderr differently so you can obviously tell. – Chris Morgan Dec 23 '10 at 00:19
or you can just do `subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)` to get them combined. Don't use `shell=True`, no reason for that. – Forrest Voight Dec 23 '10 at 00:43
@Forrest Edited that bit in. I removed `shell=True` in the process of incorporating Chris' answer. – moinudin Dec 23 '10 at 00:51
4

Under Python3, `output` won't be a string, but a bytes sequence. Try `output.decode(encoding='utf-8')` or `output.decode(encoding='latin-1')` to obtain a string. – Joachim W Sep 13 '14 at 15:53
So why is this better than the below answer? – swade Nov 07 '17 at 14:58
@swade After trying both, I learned it's because this answer allows you to capture and suppress both the stdout and stderr. Some programs output to one or the other or both. I was unable to get the below answer to suppress output. – Michael K Aug 25 '21 at 19:36

amicitas · Answer 2 · 2014-05-02T02:11:27.940

47

If you are using python 2.7 or later, the easiest way to do this is to use the subprocess.check_output() command. Here is an example:

output = subprocess.check_output('ls')

To also redirect stderr you can use the following:

output = subprocess.check_output('ls', stderr=subprocess.STDOUT)

In the case that you want to pass parameters to the command, you can either use a list or use invoke a shell and use a single string.

output = subprocess.check_output(['ls', '-a'])
output = subprocess.check_output('ls -a', shell=True)

edited May 02 '14 at 02:11

answered Mar 16 '12 at 18:25

amicitas

13,053
5
38
50

3

This does not suppress output, as requested ("I do not want the command's output to be printed to the terminal"). – Emre Jul 23 '15 at 01:09
This does suppress all output to the terminal for me (as long as I use the stderr=subprocess.STDOUT to capture both STDOUT and STDERR). Do you have an example command where it does not work? – amicitas Jul 31 '15 at 21:50
Emre, I have seen this behavior with python 2.6 – Yuriy Vasylenko Sep 26 '18 at 15:22
This also does not suppress output for me (using Python 3.9 and running an OpenSSL command) – Michael K Aug 25 '21 at 18:57

score 15 · Answer 3 · answered Dec 23 '10 at 00:17

With a = subprocess.Popen("cdrecord --help",stdout = subprocess.PIPE), you need to either use a list or use shell=True;

Either of these will work. The former is preferable.

a = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE)

a = subprocess.Popen('cdrecord --help', shell=True, stdout=subprocess.PIPE)

Also, instead of using Popen.stdout.read/Popen.stderr.read, you should use .communicate() (refer to the subprocess documentation for why).

proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()

Pipe subprocess standard output to a variable

3 Answers3

Linked

Related