Python, Popen and select - waiting for a process to terminate or a timeout

Question

I run a subprocess using:

  p = subprocess.Popen("subprocess", 
                       stdout=subprocess.PIPE, 
                       stderr=subprocess.PIPE, 
                       stdin=subprocess.PIPE)

This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.

I tried this:

  SECONDS_TO_WAIT = 10
  select.select([], 
                [p.stdout, p.stderr], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

but it just returns:

  ([],[],[])

on either condition. What can I do?

note: the subprocess might deadlock if it produces enough output. You need to consume stdout/stderr if you use PIPE — jfs, Nov 11 '12 at 20:49

score 15 · Answer 1 · edited Nov 28 '14 at 16:31

15

Have you tried using the Popen.Poll() method. You could just do this:

p = subprocess.Popen("subprocess", 
                   stdout=subprocess.PIPE, 
                   stderr=subprocess.PIPE, 
                   stdin=subprocess.PIPE)

time.sleep(SECONDS_TO_WAIT)
retcode = p.poll()
if retcode is not None:
   # process has terminated

This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.

Edit:

How about:

t_nought = time.time()
seconds_passed = 0

while(p.poll() is not None and seconds_passed < 10):
    seconds_passed = time.time() - t_nought

if seconds_passed >= 10:
   #TIMED OUT

This has the ugliness of being a busy wait, but I think it accomplishes what you want.

Additionally looking at the select call documentation again I think you may want to change it as follows:

SECONDS_TO_WAIT = 10
  select.select([p.stderr], 
                [], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).

I hope this helps.

edited Nov 28 '14 at 16:31

Alejo Dev

2,290
4
29
45

answered Dec 03 '08 at 17:06

grieve

13,220
10
49
61

Thanks for your reply, but unfortunately I can't use that method as the failure case is the most common. The program would expect approximately 600 failures (tweaking parameters each time) and then at the end, one success. Currently, I'm using `commands.getstatusoutput` but on success it will hang. – Dec 03 '08 at 17:09
Edited my answer to take your specific use case into account. – grieve Dec 03 '08 at 17:35
`if retcode:` fails if the subprocess completed successfully i.e., for `retcode == 0`; you could use `if retcode is not None:` – jfs Nov 11 '12 at 20:20
@J.F.Sebastian: Good catch! Edited to fix that. – grieve Nov 12 '12 at 14:27

score 7 · Answer 2 · edited Jul 19 '12 at 20:40

This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.

def runCmd(cmd, timeout=None):
    '''
    Will execute a command, read the output and return it back.

    @param cmd: command to execute
    @param timeout: process timeout in seconds
    @return: a tuple of three: first stdout, then stderr, then exit code
    @raise OSError: on missing command or if a timeout was reached
    '''

    ph_out = None # process output
    ph_err = None # stderr
    ph_ret = None # return code

    p = subprocess.Popen(cmd, shell=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)
    # if timeout is not set wait for process to complete
    if not timeout:
        ph_ret = p.wait()
    else:
        fin_time = time.time() + timeout
        while p.poll() == None and fin_time > time.time():
            time.sleep(1)

        # if timeout reached, raise an exception
        if fin_time < time.time():

            # starting 2.6 subprocess has a kill() method which is preferable
            # p.kill()
            os.kill(p.pid, signal.SIGKILL)
            raise OSError("Process timeout has been reached")

        ph_ret = p.returncode


    ph_out, ph_err = p.communicate()

    return (ph_out, ph_err, ph_ret)

score 3 · Answer 3 · edited Oct 06 '20 at 08:48

3

Here is a nice example:

from threading import Timer
from subprocess import Popen, PIPE

proc = Popen("ping 127.0.0.1", shell=True)
t = Timer(60, proc.kill)
t.start()
proc.wait()

edited Oct 06 '20 at 08:48

Josip

6,677
9
26
27

answered May 10 '11 at 20:03

Evan

31
1

score 2 · Answer 4 · answered Nov 11 '12 at 20:44

Python 3.3

import subprocess as sp

try:
    sp.check_call(["/subprocess"], timeout=10,
                  stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
except sp.TimeoutError:
    # timeout (the subprocess is killed at this point)
except sp.CalledProcessError:
    # subprocess failed before timeout
else:
    # subprocess ended successfully before timeout

See TimeoutExpired docs.

score 2 · Answer 5 · answered Dec 03 '08 at 17:52

Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:

from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor    

def stop(r):
    reactor.stop()
def eb(reason):
    reason.printTraceback()
def cb(result):
    stdout, stderr, exitcode = result
    # do something
getProcessOutputAndValue('/bin/someproc', []
    ).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()

Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:

http://twistedmatrix.com/projects/core/documentation/howto/process.html

What's the difference between using the code in this example and using a full ProcessProtocol? — Wolkenarchitekt, Sep 14 '11 at 07:26

score 1 · Answer 6 · answered Dec 04 '08 at 05:19

If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?

from threading import Timer
import subprocess

WAIT_TIME = 10.0

def check_cmd(cmd):
    p = subprocess.Popen(cmd,
        stdout=subprocess.PIPE, 
            stderr=subprocess.PIPE)
    def _check():
        if p.poll()!=0:
            print cmd+" did not quit within the given time period."

    # check whether the given process has exited WAIT_TIME
    # seconds from now
    Timer(WAIT_TIME, _check).start()

check_cmd('echo')
check_cmd('python')

The code above, when run, outputs:

python did not quit within the given time period.

The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.

score 0 · Answer 7 · answered Mar 11 '13 at 15:36

This is a paraphrase on Evan's answer, but it takes into account the following :

Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(

There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).

  DEVNULL = open(os.devnull, "wb")
  process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout

  def kill_process():
  """ Kill process helper"""
  try:
     process.kill()
   except OSError:
     pass  # Swallow the error

  timer = Timer(timeout_in_sec, kill_process)
  timer.start()

  process.wait()
  timer.cancel()

Python, Popen and select - waiting for a process to terminate or a timeout

7 Answers7

Python 3.3

Linked