1

I have the following python code, which runs fine on Linux, but stalls forever waiting for feedback on Windows. I am running Python 2.6.6 on Linux, python 2.7.2 on Windows, and am using the tee program from Cygwin-x86_64. If the problem doesn't seem to be reproducible on someone else's windows build, I can give more details about my setup. Here is the script:

from __future__ import with_statement

# from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python

import sys
from subprocess import PIPE, STDOUT
import subprocess, time
from threading  import Thread
import os, subprocess
import re, time
from Queue import Queue, Empty

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter((lambda : out.read(1)), b''):
        # print('0: %s' % line)
        queue.put(line)
    out.close()

class Popen_async(object):
    """Allows non-blocking reading from a Popen, by

    p = Popen_async(['myprogram.exe'], stdout=PIPE, stderr=PIPE)
    try:  line = p.stdout.get_nowait() # or p.stdout.get(timeout=.1)
    except Empty:
        print('no output yet')
    else: # got line
    """
    def __init__(self, *args, **kwargs):
        self._p = subprocess.Popen(*args, bufsize=1, close_fds=ON_POSIX, **kwargs)
        self.stdout = Queue()
        self.stderr = Queue()
        self._tout = Thread(target=enqueue_output, args=(self._p.stdout, self.stdout))
        # self._terr = Thread(target=enqueue_output, args=(self._p.stderr, self.stderr))
        self._tout.daemon = True # thread dies with the program
        # self._terr.daemon = True # thread dies with the program
        self._tout.start()
        # self._terr.start()
        self.stdin = self._p.stdin
        time.sleep(0.1)

def get_all_nowait_iter(q):
    try:
        while True:
            yield q.get_nowait()
    except Empty:
        pass

def get_all_nowait(q):
    return ''.join(get_all_nowait_iter(q))

def get_all_semiwait_iter(q):
    def log_and_return(val):
        print(val)
        return val
    try:
        # this is blocking; TODO(jgross): Figure out how to get coqtop
        # to tell us if it's finished computing
        yield log_and_return(q.get(True))
        while True:
            yield log_and_return(q.get(True, 0.1))
    except Empty:
        pass

def get_all_semiwait(q):
    return ''.join(get_all_semiwait_iter(q))

def run(filename='log'):
    p = Popen_async(['tee', filename], stdout=PIPE, stderr=STDOUT, stdin=PIPE)
    time.sleep(1)
    for i in range(10):
        print('Write: %s' % (str(i) + '\n'))
        p.stdin.write(str(i) + '\n')
        print('Wait to read...')
        stdout = get_all_semiwait(p.stdout)
        print('Read: %s' % stdout)

if __name__ == '__main__':
    run()

What's the cause of the difference in behavior between Windows and Linux?

Jason Gross
  • 5,928
  • 1
  • 26
  • 53
  • 1
    The problem goes away, apparently, if I do p.stdin.flush() after writing. I'm still curious why this occurs (and, to a lesser extent, if it's a bug in python that should be reported). – Jason Gross Jun 06 '14 at 14:30
  • Sort of a tangent, but if you have any concern that there might be a bug you should first try to reproduce it in the current version (2.7.7 for the 2.x series). – nobody Jun 06 '14 at 15:15
  • You can do an experiment to confirm that the tee process does not even receive what you write to p.stdin until you flush. You can replace tee by another process, say a simple python or java program, which is guaranteed to cause an externally visible effect, e.g. create a file as soon as it receives something from its stdin. We will then atleast know where things are stuck. – Abhishek Anand Jun 06 '14 at 15:16
  • I found that my java code for communicating with external processes also does the flushing: http://github.com/aa755/nbcoq/blob/master/src/coq/CoqTopXMLIO.java#L66 I don't remember why I did that. I will try to see what happens if I remove that line. May be this is an instance of "undefined behaviour" where an implementation may chose to buffer (not write to the stdin of the process until flush is called). – Abhishek Anand Jun 06 '14 at 15:23
  • I confirm that Java behaves in the same way. When I remove this line https://github.com/aa755/nbcoq/blob/master/src/coq/CoqTopXMLIO.java#L66 , things work fine in linux but not in windows. I usually flush after writes anyways. I think the OSes and language implementations have some flexibility about when to buffer (i guess for efficiency reasons) – Abhishek Anand Jun 06 '14 at 16:54

0 Answers0