Subprocess and using the output file for another python script within Python script

Question

Using a single python script, I want the output of one program to be captured and consumed by a second program running in a separate process. I'm attempting to use subprocesses to run these two programs from within a single python script:

prog1 = 'python prog1.py'
args1 = 'input_1.txt'
proc1 = Popen(prog1 + ' ' + args1,stdout=PIPE,stderr=STDOUT, shell=True)
for i in proc1.stdout:
   print i
prog2 = 'python prog2.py'
args2 = i   # the input which is the output of prog1
proc2 = Popen(prog2 + ' ' + args2,stdout=PIPE,stderr=STDOUT, shell=True)
for m in proc2.stdout:
    print m

Do you mean [`prog1 input_1.txt | prog2`](http://stackoverflow.com/q/295459/4279) or do you mean [`prog2 <(prog1 input_1.txt)`](http://stackoverflow.com/a/28840955/4279)? — jfs, Jul 28 '15 at 20:07

score 2 · Answer 1 · answered Jul 28 '15 at 19:26

2

Sounds like you want shell pipeline https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

p1 = Popen(["python", "prog1.py"], stdout=PIPE)
p2 = Popen(["python", "prog2.py"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
output = p2.communicate()[0]

answered Jul 28 '15 at 19:26

user590028

11,364
3
40
57

thanks for your respond. prog2.py expects an input to be pass on the command line python prog2.py input2.txt and thus I got an error filename = sys.argv[1] IndexError: list index out of range from prog2.py – BioInformatician Jul 28 '15 at 23:11

score 1 · Answer 2 · edited May 23 '17 at 10:26

If prog2 accepts input only as a filename i.e., if you need to emulate the bash process substitution: prog2 <(prog1 input_1.txt) then to avoid writing the output of prog1 subprocess to disk, you could use /dev/fd/N special filenames if your OS supports them:

#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE

prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=PIPE)
with prog1.stdout:
    cmd2 = [sys.executable, "prog2.py", "/dev/fd/%d" % prog1.stdout.fileno()]
    prog2 = Popen(cmd2, stdout=PIPE)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

You should check whether prog2 supports a special filename '-' to read the input from stdin i.e., to emulate the pipeline: prog1 input_1.txt | prog2 -:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

prog2 = Popen([sys.executable, "prog2.py", "-"], stdin=PIPE, stdout=PIPE)
prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=prog2.stdin)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

Or (better) import the modules instead of running them as subprocesses and run the corresponding functions, use multiprocessing if necessary. See Call python script with input with in a python script using subprocess

Subprocess and using the output file for another python script within Python script

2 Answers2