Logging output of file and process time out with python

Question

I am going through a big list of files and running a command on all of them. I want to log the output and if the command takes more than 5 minutes for a file, I would like to stop the command and go to the next file.

I have two problems:

1. I want to record the file name to the output file and also record the output messages. I am using Popen to log the messages and using communicate so it gets logged, but all the filenames I am writing with write() don't get written till the whole task is done.

2. I am not sure how I can poll the process and quit it after 5 minutes and go to the next file.

Below is simplified code:

import os, fnmatch
import subprocess
import sys
f=open('filenames','w')

'Locate all files matching supplied filename pattern in and below supplied root directory.'''

def locate(pattern, root=os.curdir):        
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename)


for filename in locate("*.dll"):
    f.write(filename)   
    #cmd defintion is changed for simplicity 
    cmd='cat %s' %filename
    p=subprocess.Popen(cmd,stdout=f)   
    p.communicate()[0]

Answer 1

Here's what I use, roughly:

from datetime import datetime
from time import time

import subprocess
import sys
import threading
import os

class Execution(threading.Thread):
  comm = None
  proc = None
  returncode = None
  stdoutdata = None
  stderrdate = None

  def __init__(self, comm):
    threading.Thread.__init__(self)
    self.comm = comm

  def run(self):
    self.proc = subprocess.Popen(self.comm, bufsize=-1, stdin=None,  
                               stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    self.proc.wait()
    (self.stdoutdata, self.stderrdata) = self.proc.communicate()
    self.returncode = self.proc.poll()

for curcommand in somecommandlist:
  ex = Execution(curcommand)
  starttime = time()
  ex.start()
  ex.join(some_timeout_in_seconds)

  wastimeout = False
  if (ex.isAlive()):
    wastimeout = True
    print("TIMEOUT OCCURRED @ %s" % (datetime.today(),))
    ex.proc.kill()

  ex.join()
  endtime = time()
  duration = endtime - starttime
  print("DURATION %s" % (duration,))

  exitcode = ex.returncode
  print("EXIT CODE: %s" % (exitcode,))
  print somepostprocessing(ex.stdoutdata,ex.stderrdata)
  sys.stdout.flush()

Basically: start the process inside a separate thread. This is necessary because the threading module gives you timeout waiting, while the subprocess module does not. Wait until the timeout; check if the thread is still alive. If the thread is live, we cause it to terminate by killing the process (here, subprocess gives you a kill() construct, which threading does not) and waiting for the thread to end on its own accord.

Keep in mind that communicate() blocks and stores the full output from the child's stderr and stdout, so this may not work if the output is very large.

Answer 2

1. Use f.flush() after f.write(filename).
2. See Using module 'subprocess' with timeout