5

I want to repeatedly send requests to process standard input and receive responses from standard output without calling subprocess multiple times. I can achieve a one-time request-response iteration using p.communicate however not to call the subprocess multiple times I need to use: process.stdout.readline() which hangs. How to use it properly? I use Python 2.7 64 bit, Windows 7. Thanks in advance.

main.py:

import subprocess
p = subprocess.Popen(['python','subproc.py'],stdin=subprocess.PIPE,stdout=subprocess.PIPE)

while True:
    s=raw_input('Enter message:')
    p.stdin.write(s)
    p.stdin.flush()
    response = p.stdout.readline()
    if response!= '':
        print "Process response:", response
    else:
        break

subproc.py:

from __future__ import division
import pyximport
s=raw_input()
print 'Input=',s
dano
  • 91,354
  • 19
  • 222
  • 219
Apogentus
  • 6,371
  • 6
  • 32
  • 33

3 Answers3

10

There are a couple of minor tweaks you can make to get this working. First is to disable buffered output in the child using the -u option. Second is to send a newline character along with the user-inputted message to the child process, so that the raw_input call in the child completes.

main.py

import subprocess

# We use the -u option to tell Python to use unbuffered output
p = subprocess.Popen(['python','-u', 'subproc.py'],
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE)

while True:
    s = raw_input('Enter message:')
    p.stdin.write(s + "\n")  # Include '\n'
    p.stdin.flush()
    response = p.stdout.readline()
    if response != '': 
        print "Process response:", response
    else:
        break

You should also wrap the child process in an infinite loop, or things will break after the first message is sent:

subproc.py:

while True:
    s = raw_input()
    print 'Input=',s

Output:

dan@dantop:~$ ./main.py 
Enter message:asdf
Process response: Input= asdf

Enter message:asdf
Process response: Input= asdf

Enter message:blah blah
Process response: Input= blah blah

Enter message:ok
Process response: Input= ok
dano
  • 91,354
  • 19
  • 222
  • 219
5

It is not safe to assume that the child process will immediately get the complete data you send to its stdin, as buffers can get in the way. If you must hold the file open for further output then you should at least invoke its flush() method.

Furthermore, it is not safe to assume that the child process's output will be immmediately available to you to read. If it does not flush (or close) its output stream then the EOL could be buffered, and if you do nothing to prompt the child process to act further, then your readline() may wait forever. But your program then CAN'T do anything, because it's stuck in readline(). If the child process is built for this then you might get it to work, but otherwise you need to use a safer method, such as subprocess.communicate().

As you observe, it does not work to call communicate() more than once on the same subprocess. That's as you should expect from its documentation: it reads all output until end-of-file, and waits for the subprocess to terminate. To send multiple inputs in this mode, build a string containing all of them, pass it to communicate() and then read all the answers.

Alternatively, if you really need to alternate between writing and reading, and your subprocess is not specifically tooled for that, then it is safer to do the reading and writing in separate threads, without any assumption of the writes and reads interlacing perfectly.

John Bollinger
  • 160,171
  • 8
  • 81
  • 157
  • Thanks for pointing out the flush thingy, that's very easy to overlook when you are just thinking about communication with another program and I debugged about every other aspect of a script of mine trying to figure out where I went wrong. Thats a really nasty source for bugs, because part of the message arrives correctly while part of it still hangs in the buffer. – logical x 2 Apr 17 '17 at 18:34
-5

You need to use communicate to interact with your sub-process (https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate). Here is an updated version of your main code:

import subprocess
p = subprocess.Popen(['python','subproc.py'],stdin=subprocess.PIPE,stdout=subprocess.PIPE)

while True:
    s = raw_input('Enter message:')
    response, _ = p.communicate(s)
    if response!= '':
        print "Process response:", response
    else:
        break

You also have to be careful that while you have a loop in your main code your subproc code only runs once. Only the first iteration will correctly receive a response the second communicate call will raise an exception as the stdin file of the subprocess will be closed by then.

qfiard
  • 869
  • 8
  • 4
  • 5
    The problem with using `p.communicate()` is that in addition to sending data via stdin, it also blocks until the child process is complete. The OP wants to be able to repeatedly send data to stdin without having to restart the child process. – dano Jul 08 '14 at 19:43