stdout.read() from finished subprocess sometimes returning empty?

Question

I have created a dictionary where I associate an id with a subprocess. Something like:

cmd = "ls"
processes[id] = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE)

Then I call a method with this process map as an input, that checks which process has finished. If the process finishes, I check the process's stdout.read() for a particular string match.

The issue is sometimes stdout.read() returns an empty value which causes issues in string matching.

Sample Code:

#Create a map
processes[id] = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE)
...
#Pass that map to a method which checks which processes have finished
completedProcesses(processes)

def completedProcesses(processes):
    processList = []
    for id,process in processes.iteritems():
        if process.poll() is not None:
            #If some error in process stdout then print id
            verifySuccessStatus(id, processes[id])
            processList.add(id)

def verifySuccessStatus(id, process):
    file=open(FAILED_IDS_FILE, 'a+')
    buffer =  process.stdout.read() #This returns empty value sometime
    if 'Error' not in buffer:
        file.write(id)
        file.write('\n')
    file.close()

I am new to python, I might be missing some internal functionality understanding of subprocess

do you mean 'error' instead of error' note the missing quotes — sam, Oct 12 '15 at 21:55
@AFH most users can't do single character edits. But you as the original poster can edit to correct it to 'Error'. — CivFan, Oct 12 '15 at 22:14
You do realise check_output will do what most of your code is doing and shell=True is not needed? — Padraic Cunningham, Oct 12 '15 at 22:53

score 1 · Answer 1 · edited May 23 '17 at 12:14

There are at least two issues:

There is no point to call process.stdout.read() more than once. .read() does not return until EOF. It returns an empty string to indicate EOF after that.
You should read from the pipes while the processes are still running otherwise they may hang if they generate enough output to fill OS pipe buffers (~65K on my Linux box)

If you want to run multiple external processes concurrently and check their output after they are finished then see this answer that shows "thread pool" and async.io solutions.

CivFan · Answer 2 · 2015-10-12T23:19:51.737

Judging by your example command of ls, your issue may be caused by the stdout pipe filling up. Using the process.communicate() method handles this case for you, since you don't need to write multiple times to stdin.

# Recommend the future print function for easier file writing.
from __future__ import print_function

# Create a map
# Keeping access to 'stderr' is generally recommended, but not required.
# Also, if you don't know you need 'shell=True', it's safer practice not to use it.
processes[id] = subprocess.Popen(
    [cmd],
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    )
...
#Pass that map to a method which checks which processes have finished
check_processes(processes)

def check_processes(processes):
    process_ids = []

    # 'id' is a built-in function in python, so it's safer to use a different name.
    for idx, process in processes.iteritems():
        # When using pipes, communicate() will handle the case of the pipe
        # filling up for you.
        stdout, stderr = process.communicate()
        if not is_success(stdout):
            write_failed_id(idx)
        process_ids.append(idx)

def is_success(stdout):
    return 'Error' not in stdout

def write_failed_id(idx):
    # Recommend using a context manager when interacting with files.
    # Also, 'file' is a built-in function in python.
    with open(FAILED_IDS_FILE, 'a+') as fail_file:
        # The future print function makes file printing simpler.
        print(idx, file=fail_file)

CivFan · Answer 3 · 2015-10-12T22:24:13.983

-1

You're only reading stdout and looking for "Error". Perhaps you should also be looking in stderr:

processes[id] = subprocess.Popen(
    [cmd], 
    shell=True, 
    stdout=subprocess.PIPE, 
    stderr=subprocess.STDOUT,
    )

From the subprocess docs:

subprocess.STDOUT

Special value that can be used as the stderr argument to Popen and indicates that standard error should go into the same handle as standard output.

The process could have failed unexpectedly, returning no stdout but a non-zero return code. You can check this using process.returncode.

Popen.returncode

The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.

A negative value -N indicates that the child was terminated by signal N (Unix only).

edited Oct 12 '15 at 22:24

answered Oct 12 '15 at 22:02

CivFan

13,560
9
41
58

Output is in stdout. I sometimes see buffer but sometimes it is empty. Is buffer kept intact even of the process gets terminated? – AFH Oct 12 '15 at 22:07
@AFH I don't know. Are you expecting the process to be terminated? That's not in the original question, so just wondering as it is an interesting possibility. Have you checked the process return code? It will tell which signal terminated it, if you're not expecting the process to be terminated. – CivFan Oct 12 '15 at 22:13

stdout.read() from finished subprocess sometimes returning empty?

3 Answers3