subprocess.Popen with shell=True hangs on 'time <...>'

Question

I'm using this code, which is supposed to be transitional for python2 to python3 porting phase (I know there are third-party libs for run, I want to implement my own to collect experience)

def run(args,
        stdin=None, input=None,
        stdout=None, stderr=None, capture_output=False,
        timeout=None,
        encoding=None,
        **popen_kwargs):        
    # create stderr and stdout pipes, if capture_output is true
    if capture_output:
        _stderr = subprocess.PIPE
        _stdout = subprocess.PIPE
    else:
        _stdout, _stderr = stdout, stderr

    # if input is given, create stdin as pipe, where input will be passed into
    if input is not None:
        _stdin = subprocess.PIPE
    else:
        _stdin = stdin

    # this starts the process. python2 did not have 'encoding'
    if sys.version_info.major >= 3:
        proc = subprocess.Popen(args, stdin=_stdin, stdout=_stdout, stderr=_stderr, encoding=encoding,
                                **popen_kwargs)
    else:
        proc = subprocess.Popen(
            args, stdin=_stdin, stdout=_stdout, stderr=_stderr, **popen_kwargs)

    # run a background timer to interrupt 'communicate', if necessary
    if timeout is not None:
        def cancel():
            try:
                proc.terminate()
            except OSError:
                # an exception here means that the process is gone already
                pass
        cancel_timer = Timer(timeout, cancel)
        cancel_timer.start()

    # special case for python2 for which we allow passing 'unicode' if an encoding is used
    if input is not None and sys.version_info.major < 3:
        if type(input) == unicode:
            import codecs
            input = codecs.encode(input, encoding)
    (stdoutoutput, stderroutput) = proc.communicate(input)

    # check timeout scenario
    if timeout is not None:
        if not cancel_timer.is_alive():
            raise TimeoutExpired(args, timeout, stdoutoutput,
                                 stdoutoutput, stderroutput)
        else:
            cancel_timer.cancel()
            cancel_timer.join()

    # on python2, outputs will always be 'str', which is fine with us, as it's the union of
    # str and bytes
    return CompletedProcess(args, proc.poll(), stdoutoutput, stderroutput)

However, the code blocks indefinitely within communicate whenever I try to execute anything with the shell builtin time prefixed and capture_output on both python2 and python3. It must be something really stupid.

>>> run("time sleep 5m", shell=True, capture_output=True, timeout=1)
... (^ C to stop it)
>>> run("sleep 5m", shell=True, capture_output=True, timeout=1)
zsubprocess.TimeoutExpired: sleep 5m: Timeout after 1 seconds

I do not understand why that is the case.

Johannes Schaub - litb · Answer 1 · 2021-03-10T13:25:30.520

As it turned out, subprocess does not execute a shell when the command is just sleep 5m, but only when it is time sleep 5m (more accurately: sh optimizes its -c and apparently execs it instead of forking).

When I executed proc.terminate(), it only terminated the wrapper shell, but not the sleep, which became a child of the child subreaper (init here). And possibly (I guess) communicate wait for the pipes to yield EOF to collect outstanding data, which won't happen unless the sleep terminates. The following solution was taken from https://stackoverflow.com/a/25134985/34509 . Instead of proc.kill, do

process = psutil.Process(proc.pid)
if proc.poll() is None:
    for child in process.children(recursive=True):
        child.kill()
    process.kill()

There's a small chance that we could race with the process, where it could create new processes before we could kill them, but at least simple cases work now!

subprocess.Popen with shell=True hangs on 'time <...>'

1 Answers1