Capturing output from Popen

Question

Maybe what I need is a general explanation of what STDOUT is, but here's my problem. I need to run a shell script within python on a bunch of pairs of files, and parse the output. If I run:

from itertools import combinations
from subprocess import Popen

for pair in combinations(all_ssu, 2):
    Popen(
        ['blastn',
        '-query', 'tmp/{0}.fna'.format(pair[0]),
        '-subject', 'tmp/{0}.fna'.format(pair[1]),
        '-outfmt', '6 qseqid sseqid pident'
        ],
    )

... it seems to work great (note: all_ssu is a list of file names essentially). The shell prints a bunch of lines of data that I'd like to compare. So how do I get that printed data into a list or a dataframe or something so that I can use it?

After looking around the docs and some other questions here, it looks like the stdout flag is looking for a file object, so I tried:

from itertools import combinations
from subprocess import Popen
for pair in combinations(all_ssu, 2):
    out_file = open('tmp.txt', 'rw')
    Popen(
        ['blastn',
        '-query', 'tmp/{0}.fna'.format(pair[0]),
        '-subject', 'tmp/{0}.fna'.format(pair[1]),
        '-outfmt', '6 qseqid sseqid pident'
        ],
        stdout=out_file
    )
    for line in out_file.readlines():
        print line
    out_file.close()

And this also seems to work, except that I create that temporary file that I don't need. I tried just setting a variable captured to None and then putting stdout=captured, but in this case it's just setting captured to 0. I also tried out = Popen(...) without the stdout flag, but again, out is just int(0). I also tried playing around with PIPE, but couldn't make heads or tails of it.

So the question is: how do I just capture the output from Popen directly?

When you use `PIPE`, you read from the pipe using `Popen.communicate`. The Python online docs do have good examples of this. — Derek T. Jones, Jul 03 '15 at 15:27

score 3 · Answer 1 · edited Aug 07 '15 at 09:21

3

From Python 2.7 onwards , you can use - subprocess.check_output

It returns the output of the executed command back as a byte string.

Example -

>>> import subprocess
>>> s = subprocess.check_output(["echo","Hello World!"], shell=True)
>>> s
b'"Hello World!"\r\n'

I had to use shell=True on my windows for this to work, but on linux this may not be needed.

edited Aug 07 '15 at 09:21

Sharon Dwilif K

1,218
9
17

answered Jul 03 '15 at 15:30

Anand S Kumar

88,551
18
188
176

Note that `subprocess.check_output` is new in Python 2.7. Usually that's not a problem, but it's good to keep in mind. – Thomas Orozco Jul 03 '15 at 15:33
Thanks, added that to the answer. – Anand S Kumar Jul 03 '15 at 15:33

score 2 · Accepted Answer · edited May 23 '17 at 12:21

Try this:

from itertools import combinations
from subprocess import Popen, PIPE
for pair in combinations(all_ssu, 2):
    out = Popen(
        ['blastn',
        '-query', 'tmp/{0}.fna'.format(pair[0]),
        '-subject', 'tmp/{0}.fna'.format(pair[1]),
        '-outfmt', '6 qseqid sseqid pident'
        ],
        stdout=PIPE
    ).communicate[0]
    print(out)

from How can I get terminal output in python?

STDOUT is just the standard output for the program, which is the file where anything that the program prints will be written to.

If you want the outputs as a list then you can just create an empty list (l = [] or something) before the loop and do l.append(out) at the end of each iteration of the loop

Capturing output from Popen

2 Answers2