Passing output from an external program back to it after processing

Question

I have a Fortran code that takes a file as input and writes the output into stdout. To avoid read/write cycles, I'd like to run the code inside python and convert the output to a numpy array. I can do this using the following function:

def run_fortran(infile):

    import subprocess
    import numpy as np

    cmd = ['./output.e', infile]
    p = subprocess.Popen(cmd, stdout = subprocess.PIPE)
    out, err = p.communicate()

    if p.returncode == 0: 
        return np.array(out.split(),dtype=int)

Now I take the array, modify it and write it into a file. The new file is again passed into run_fortran(infile). Can I avoid this step and somehow use the output of run_fortran instead of passing a filename?

I tried two cases with no success:

(1) converting to string:

arr = run_fortran('input.txt')
new_arr = str(arr).replace("[","").replace("]","")
run_fortran(new_arr)

This returns an empty array.

(2) converting to a file type object using StringIO:

from cStringIO import StringIO
run_fortran(StringIO(new_arr))

This returns an error: TypeError: execv() arg 2 must contain only strings which makes sense.

FYI, you can use `check_output` instead of `Popen`+`communicate`. — ivan_pozdeev, Oct 24 '16 at 01:25
your fortran program expects file name so it will no work with anything else. if you change fortran program to work with data from command line (or keyboard - stdin) then you can use it without saving in file. — furas, Oct 24 '16 at 01:42
@furas: That makes sense, thank you. I need to figure out a way to do that in Fortran! — Mahdi, Oct 24 '16 at 01:46
some programs (created for Linux) can run in two ways: if you run with argument then it uses it as file name, if you run without argument then it reads from keyboard (stdin) and then you can use data directly from other program using "pipe" `|` - ie. `one_program | your_program | other_program > results.txt` — furas, Oct 24 '16 at 02:05
you can directly link the fortran code into your python routine using f2py: https://docs.scipy.org/doc/numpy-dev/f2py/usage.html , there is no need to go via shell — dnalow, Oct 24 '16 at 09:36

score 0 · Answer 1 · answered Oct 24 '16 at 23:56

In fortran the read(*,*), read(5,*) or read*, statement will read from from standard input, and if that has the right format, it will work. If you are dealing with formatted data, that is, anything that is human readable, and not a binary file, then you probably need a read loop like:

do line=1,numlines
  read(*,*) array(line,:)
enddo

No open or close statements needed. So if what you were writing to a file is passed directly, you should be able to remove the those statements and change the file unit to 5 or *.

Now there are more efficient ways to do this kind of communication, but any solution is a good solution if it suit your purpose.

ivan_pozdeev · Answer 2 · 2016-10-27T20:46:04.637

If your Fortran program (it's that './output.e' AFAICS) can read from stdin, not only from a regular file, you can do without temporary files by passing it stdin=subprocess.PIPE (other valid values are "an existing file descriptor (a positive integer) [or] an existing file object"). In UNIX, there's always /dev/stdin to put on the command line btw and in Windows, there's con.

Still, if the program can only work in "sessions" due to the nature of processing (i.e. cannot run continuously and be fed new data as it's available), you'll have to invoke it repeatedly.

Note that you have to handle different pipes in different threads to avoid deadlocks. So, either use communicate() (but then the program cannot run continuously) or spawn an stdout/stderr thread manually (that's what communicate() does; not stdin because the output reading code has to be running by the time you start writing to stdin or the external program may choke with "no space left on device" while writing).

Here's sample code for running the program continuously:

p=subprocess.Popen(argv,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
while True:
    #the only more elegant way to return a value from a thread is with a Thread subclass,
    # see http://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread-in-python
    output_container=[]
    ot=threading.Thread(target=_outputreaderthread,args=(p.stdout,output_container,data))
    ot.start()
    p.stdin.write(data)
    ot.join()
    output=output_container[0]

    data=process_output(output)
    if no_more_processing_needed(data): break
p.stdin.close()
if p.wait()!=0:raise subprocess.CalledProcessError(p.returncode,argv)

def _outputreaderthread(stream,container,data):
    #since there's no process termination as an end mark here,
    # you need to read exactly the right number of bytes to avoid getting blocked.
    #use whatever technique is appropriate for your data (e.g. readline()'s)
    output=stream.read(calculate_output_size(data))
    container.append(output)

Passing output from an external program back to it after processing

2 Answers2