python subprocess continuously feed stdin and read stdout which is possibly empty

Question

I am trying to write a python script that optimises the parameters of a compiled c script that I wrote. The program (dups) is taking an input (using fgets) and prints the input if it is not in the set of previously seen inputs.

My python script should continuously feed input to dups followed by an evaluation of it's output (which can be either the input or nothing).

I am using subprocess writing to its stdin and reading from its stdout. However, using various solutions, the script hangs after reading the first line.

The C program only writes to stdout and I know that it is flushing. Furthermore I am able to reproduce my error using a python script simulating dups.

Sample code:

Sample dups for easy reproduction:

#!/usr/bin/python
import sys
names = set()
while True:
    try:
        x = raw_input()
    except EOFError:
        print "Unexpected EOF, quitting"
        break
    if x not in names:
        print x
        sys.stdout.flush()
    names.add(x)

Main script:

import subprocess
import pty, os #for solution 4

inputs = ['Alice', 'Alice', 'Bob', 'Jane', 'Bob', 'Jane', 'Alice', 'Mike']

solution = 4

arguments = ['./sample_script.py']

popen_nargs = dict(stdin=subprocess.PIPE,
                   stdout=subprocess.PIPE,
                   universal_newlines=True,
                   )

if solution == 3:
    out_fname = "output.txt"
    f_w = open(out_fname,"wt")
    f_r = open(out_fname, "rt")
    popen_nargs['stdout'] = f_w
    popen_nargs['bufsize']=0

if solution == 4:
    master, slave = pty.openpty()
    popen_nargs['stdout'] = slave
    stdout_handle = os.fdopen(master)

child = subprocess.Popen(arguments, **popen_nargs)

for inpt in inputs:

    print "Testing '" + inpt + "'"
    child.stdin.write(inpt +" \n")
    child.stdin.flush()


    print "Result:"

    #http://stackoverflow.com/questions/4417546/constantly-print-subprocess-output-while-process-is-running
    if solution == 0:
        for stdout_line in iter(child.stdout.readline, ""):
            print "-> '", stdout_line, "'"

    #http://blog.endpoint.com/2015/01/getting-realtime-output-using-python.html
    elif solution == 1:
        while True:
            output = child.stdout.readline()
            if output == '' and child.poll() is not None:
                break
            if output:
                print "-> '", output, "'"

    #https://gist.github.com/zhenyi2697/7229421
    elif solution == 2:
        output=''
        while True:
            result = child.poll()
            delta = child.stdout.read(1)

            if result is not None:
                print 'terminated'
                break

            if delta != ' ':
                output = output + delta
            else:
                if '%' in output:
                    print 'percentage is:'
                    print output

                elif '/s' in output:
                    print 'speed is:'
                    print output

                print "-> '", output, "'"
                output = ''

    #http://stackoverflow.com/questions/5419888/reading-from-a-frequently-updated-file
    elif solution == 3:
        f_w.flush()
        print "-> '", f_r.read(), "'"

    print "end iteration"

    #http://stackoverflow.com/questions/13603334/repeatedly-write-to-stdin-and-read-stdout-of-a-subprocess-without-closing-it
    if solution == 4:
        print "-> '", stdout_handle.readline(), "'"

if solution == 3:
    f_w.close()
    f_r.close()

# Close the program
child.communicate()

Output:

• Solutions 0, 1 (don't terminate):

  Testing 'Alice'
  Result:
  -> ' Alice 
  '
  

• Solution 2 (doest terminate):

  Testing 'Alice'
  Result:
  -> ' Alice '
  

• Solution 3 (output.txt seems to be updated only in the end, despite bufsize=0):

  Testing 'Alice'
  Result:
  -> '  '
  end iteration
  Testing 'Alice'
  Result:
  -> '  '
  end iteration
  
  ...
  
  Testing 'Mike'
  Result:
  -> '  '
  end iteration
  

• Solution 4 (seems promising, but read and readline do not work when there is no output hence it does not terminate):

  Testing 'Alice'
  Result:
  -> ' Alice 
  '
  end iteration
  Testing 'Alice'
  Result:
  -> '