Python - How to do non-blocking read from PIPE in subprocess.Popen?

Question

I've posted similar question last week and this post reflects my trial and the problem I am facing now.

The program to invoke through Popen is a command line program. I use one thread to read one item from queue and send it to the stdin and get the response from stdout. However it hangs in the proc.stdout.read(). I did see it works fine with expected output last Friday and then when I did some changes today, it hangs. The change I made is replace read() with readlines() and use a loop to iterate the results. I am aware that readlines() would probably block but when I reversed the code to where it was with read() last Friday, it blocks, too. I completely get lost to it now. Any possible causes?

Below is the code to get one sentence from queue and feed it to the java program to get a response:

''' below is the code for worker thread. '''
def readQueue(proc, queue):
    print 'enter queueThread.\n'
    global notEmpty
    notEmpty = True
    while notEmpty:
        try:
            sen = queue.get()
            proc.stdin.write(sen.strip())
            res = proc.stdout.read()
            print res.strip(), ' ', sen
            queue.task_done()
        except Empty:
            break
    print 'leave queueThread.'

The main thread below is to read each line from a file and put it in a queue for the worker thread to process item by item:

def testSubprocess():
    ee = open('sentences.txt', 'r')
    #ff = open('result.txt', 'w')   # print it to stdout first before really write to a file. 
    lines = ee.readlines()
    cmd = ['java', 
           '-cp', 'someUsefulTools.jar', 
           'className', 
           '-stdin',]   # take input from stdin
    proc = Popen(cmd, stdout=PIPE, stdin=PIPE, stderr=PIPE, bufsize=-1, universal_newlines=True)
    q = Queue()
    for sen in lines:
        q.put(sen.strip())
    readThread = Thread(target=readQueue, args=(proc, q))
    readThread.daemon = True
    readThread.start()
    print 'Main thread is waiting...\n'
    q.join()
    global notEmpty; notEmpty = False
    print 'Done!'

Apologies if I have misunderstood your code and this doesn't work, however I found that I can stop a Pipe from blocking this way. (I don't have a lot of experience with threading in python) After you define the "proc" add this: "fcntl.fcntl(proc, fcntl.F_SETFL, os.O_NONBLOCK)" if proc is being defined as type "pipe" this should stop it blocking. you may need to import fcntl. — Aphire, Apr 07 '15 at 11:13
@Aphire not sure if I got it right, it seemed "fcntl" is used for linux/unix environment, right? (my working environment is windows) — Frank Yang, Apr 08 '15 at 03:20
Imagine you send a sentence to the java program: how do you know when to stop reading the response: is it *always* 10 bytes or is it *always* a single line or do you want to read until EOF i.e., each process can answer only a **single** question? — jfs, Apr 08 '15 at 20:23

score 1 · Answer 1 · answered Apr 07 '15 at 13:30

1

The pipes from the subprocess are file like objects.

The method read() on these objects reads everything into memory until EOF is reached if you don't specify how much is should read, see doc: https://docs.python.org/2/library/stdtypes.html#file.read.

You have to set the size you want to read if you don't want this behaviour. Try setting it to 1 byte?

The same goes for readlines(), see documentation: https://docs.python.org/2/library/stdtypes.html#file.readlines

answered Apr 07 '15 at 13:30

J. P. Petersen

4,871
4
33
33

Have tried and set the size to read. Problem remains the same. Almost want to give up and turn to [pexpect](https://pexpect.readthedocs.org/en/latest/). – Frank Yang Apr 08 '15 at 03:27
Some programs might behave differently when executed through `pexpect` as the programs can detect if they are running in a terminal (TTY device). Don't know if Java programs can do this. For example if I run `ps aux` on my system, `ps` detects that the terminal is only 80 characters wide, and shortens the lines accordingly. If we pipe the output into a file (like with subprocess), it will not know what the width of the terminal is, and therefor not shorten the lines. – J. P. Petersen Apr 08 '15 at 10:55

Python - How to do non-blocking read from PIPE in subprocess.Popen?

1 Answers1