python log stderr and stdout to a file

Question

Just wanted to pick your brains on this. I have several subprocesses running and I wanted to print the stderr and stdout to a file. I've done this so far:

def write_to_stderr_log(process):
    stderr= open("stderr.log", "w")
    proc_err = process.communicate()
    print >> stderr, proc_err
    stderr.close()

def write_to_stdout_log(process):
    stdout = open("stdout.log", "w")
    proc_out = process.communicate()
    print >> stdout, proc_out
    stdout.close()

def logger():
    logger = logging.getLogger('error_testing')
    hdlr = logging.FileHandler('error.log')
    formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
    hdlr.setFormatter(formatter)
    logger.addHandler(hdlr)
    logger.setLevel(logging.WARNING)

    logger.error('We have a problem')
    logger.debug('debugging')
    logger.info('some info')

logger()
proc = subprocess.Popen(['FastTree -nt test.fasta'], bufsize=512, stdin = None, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True) 

write_to_stderr_log(proc)
write_to_stdout_log(proc)

Is this the best way to do this. If I have more than one process, I guess its going to re-write the log files, so that could be a problem. Appreciate some advice here. Thanks

@nrpeterson this one is about redirecting the output of a child process — D.Shawley, Jun 13 '13 at 11:48

score 0 · Answer 1 · answered Jun 13 '13 at 11:47

I'm pretty sure that you can provide your own file instances as keywords to subprocess.Popen.

>>> out = open('stdout.log', 'wb')
>>> err = open('stderr.log', 'wb')
>>> child = subprocess.Popen('FastTree -nt test.fasta', stdin=None, stdout=out,
                             stderr=err)
>>> rc = child.wait()
>>> out.close()
>>> err.close()

Here's the portion of the subprocess.Popen documents that matters:

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout.

That hasnt worked. If I introduce an error on the 'FastTree' subprocess, nothing is printed to the stderr.log file. — bioinf80, Jun 13 '13 at 15:12
Try adding `shell=True` though I doubt that will have any effect. — D.Shawley, Jun 13 '13 at 23:14

python log stderr and stdout to a file

1 Answers1