Keeping a pipe to a process open

Question

I have an app that reads in stuff from stdin and returns, after a newline, results to stdout

A simple (stupid) example:

$ app
Expand[(x+1)^2]<CR>
x^2 + 2*x + 1
100 - 4<CR>
96

Opening and closing the app requires a lot of initialization and clean-up (its an interface to a Computer Algebra System), so I want to keep this to a minimum.

I want to open a pipe in Python to this process, write strings to its stdin and read out the results from stdout. Popen.communicate() doesn't work for this, as it closes the file handle, requiring to reopen the pipe.

I've tried something along the lines of this related question: Communicate multiple times with a process without breaking the pipe? but I'm not sure how to wait for the output. It is also difficult to know a priori how long it will take the app to finish to process for the input at hand, so I don't want to make any assumptions. I guess most of my confusion comes from this question: Non-blocking read on a subprocess.PIPE in python where it is stated that mixing high and low level functions is not a good idea.

EDIT: Sorry that I didn't give any code before, got interrupted. This is what I've tried so far and it seems to work, I'm just worried that something goes wrong unnoticed:

from subprocess import Popen, PIPE
pipe = Popen(["MathPipe"], stdin=PIPE, stdout=PIPE)         
expressions = ["Expand[(x+1)^2]", "Integrate[Sin[x], {x,0,2*Pi}]"] # ...                                                                                        
for expr in expressions:
    pipe.stdin.write(expr)                                                        
    while True:                                                                         
        line = pipe.stdout.readline()                                                    
        if line != '':     
            print line  
        # output of MathPipe is always terminated by ';'                                                                                                                         
        if ";" in line:                                                                  
            break

Potential problems with this?

While this looks like Mathematica, I asked a similar question earlier for MAPLE. Take a look at http://stackoverflow.com/questions/2053231/grabbing-the-output-of-maple-via-python — Hooked, Jun 14 '11 at 17:08
@Hooked: Thanks for this. It looks like what I'm doing is in one of the answers to your question... — bbtrb, Jun 14 '11 at 17:37

score 3 · Accepted Answer · answered Jun 14 '11 at 16:24

3

Using subprocess, you can't do this reliably. You might want to look at using the pexpect library. That won't work on Windows - if you're on Windows, try winpexpect.

Also, if you're trying to do mathematical stuff in Python, check out SAGE. They do a lot of work on interfacing with other open-source maths software, so there's a chance they've already done what you're trying to.

answered Jun 14 '11 at 16:24

Thomas K

39,200
7
84
86

Thomas, can you please explain why subprocess is not "reliable" in this respect? – jforberg Jun 16 '11 at 14:35
@jforberg: Well, the only way I know to use subprocess to repeatedly interact with a process is a platform specific hack. It also relies on using sp.stdout.read() and sp.stdin.write(), which the subprocess docs specifically warn you not to do. – Thomas K Jun 16 '11 at 16:51

jforberg · Answer 2 · 2011-06-15T08:37:24.273

2

Perhaps you could pass stdin=subprocess.PIPE as an argument to subprocess.Popen. This will make the process' stdin available as a general file-like object:

import sys, subprocess

proc = subprocess.Popen(["mathematica <args>"], stdin=subprocess.PIPE, 
                        stdout=sys.stdout, shell=True)
proc.stdin.write("Expand[ (x-1)^2 ]") # Write whatever to the process
proc.stdin.flush()                    # Ensure nothing is left in the buffer
proc.terminate()                      # Kill the process

This directs the subprocess' output directly to your python process' stdout. If you need to read the output and do some editing first, that is possible as well. Check out http://docs.python.org/library/subprocess.html#popen-objects.

edited Jun 15 '11 at 08:37

answered Jun 14 '11 at 17:05

jforberg

6,537
3
29
47

Would `proc.wait()` not wait for the subprocess to finish completely? – bbtrb Jun 14 '11 at 17:38
True, I've changed my answer to say terminate() instead. Perhaps it would be better to send an exit command through the pipe, but I don't know how this program works. – jforberg Jun 15 '11 at 08:52

Keeping a pipe to a process open

2 Answers2

Linked