returncode of Popen object is None after the process is terminated

Question

I'm running a process with the use of Popen. I need to wait for the process to terminate. I'm checking that the process have terminated through the returncode. When returncode is different from None the process must have terminated. The problem is that when print_output is False the returncode is always None, even when the process have finished running (terminated). This is however not the case when print_output is True. I'm using the following code to run the process:

def run(command, print_output=True):
    # code mostly from: http://sharats.me/the-ever-useful-and-neat-subprocess-module.html
    from subprocess import Popen, PIPE
    from threading import Thread
    from queue import Queue, Empty
    from time import sleep

    io_q = Queue()

    def stream_watcher(identifier, stream):
        for line in stream:
            io_q.put((identifier, line))

        if not stream.closed:
            stream.close()

    with Popen(command, stdout=PIPE, stderr=PIPE, universal_newlines=True) as proc:
        if print_output:

            Thread(target=stream_watcher, name='stdout-watcher', args=('STDOUT', proc.stdout)).start()
            Thread(target=stream_watcher, name='stderr-watcher', args=('STDERR', proc.stderr)).start()

            def printer():
                while True:
                    try:
                        # Block for 1 second.
                        item = io_q.get(True, 1)
                    except Empty:
                        # No output in either streams for a second. Are we done?
                        if proc.poll() is not None:
                            break
                    else:
                        identifier, line = item
                        print(identifier + ':', line, end='')

            Thread(target=printer, name='printer').start()

        while proc.returncode is None:
            sleep(2)
            proc.poll()

        if not proc.returncode == 0:
            raise RuntimeError(
                'The process call "{}" returned with code {}. The return code is not 0, thus an error '
                'occurred.'.format(list(command), proc.returncode))

        return proc.stdout, proc.stderr

Any clues to what might cause this problem?

EDIT: Discovered something pretty weird. I'm running the following code:

run(my_command, True)
print('--------done--------')
run(my_command, False)
print('--------done--------')

'--------done--------' is never printed even though run(my_command, False) gets executed.

Answer 1

TL;DR

add popen.wait() after subprocess.Popen()

Explanation Part (sort of)

Python goes too fast and the child process is ended but returncode can't be read

(I don't really know why it does that. Explanations welcome)

Why did I use this:

Shell command execution and get both return code and output (stdout)

def exec_cmd(cmd):
    pop = subprocess.Popen(shlex.split(cmd), stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
    pop.wait()
    return [pop.returncode, pop.communicate()[0]]

Also: please read the .wait warning on the popen page

Answer 2

I'm not sure why it did not work, but I think it has something to do with not closing the streams. The following code works:

def run(command, print_output=True):
    from subprocess import Popen, PIPE, STDOUT
    from io import StringIO

    popen = Popen(command, stdout=PIPE, stderr=STDOUT, universal_newlines=True)
    out = StringIO()
    for line in popen.stdout:
        if print_output:
            print(line, end='')
        else:
            out.write(line)

    popen.stdout.close()
    return_code = popen.wait()

    if not return_code == 0:
        raise RuntimeError(
            'The process call "{}" returned with code {}. The return code is not 0, thus an error '
            'occurred.'.format(list(command), return_code))

    stdout_string = out.getvalue()
    out.close()

    return stdout_string