python script using subprocess, redirect ALL output to file

Question

I am writing something for static analysis of source code in different languages. As anything has to be open source and callable from command line I now have downloaded one tool per language. So I decided to write a python script listing all source files in a project folder and calling the respective tool.

So part of my code looks like this:

import os
import sys
import subprocess
from subprocess import call
from pylint.lint import Run as pylint


class Analyser:

    def __init__(self, source=os.getcwd(), logfilename=None):

        # doing initialization stuff
        self.logfilename = logfilename or 'CodeAnalysisReport.log'

        self.listFiles()
        self.analyseFiles()


    def listFiles(self):
    # lists all source files in the specified directory


    def analyseFiles(self):

        self.analysePythons()
        self.analyseCpps()
        self.analyseJss()
        self.analyseJavas()
        self.analyseCs()


if __name__ == '__main__':

    Analyser()

Let's have at a look at the C++ files part (I use Cppcheck to analyse those):

    def analyseCpps(self):

        for sourcefile in self.files['.cc'] + self.files['.cpp']:
            print '\n'*2, '*'*70, '\n', sourcefile
            call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile])

The console output for one of the files (it's just a random downloaded file) is:

**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
(information) Cppcheck cannot find all the include files (use --check-config for details)

Line 1 and 2 coming from my script, lines 3 to 7 coming from Cppcheck.

And this is what I want to save to my log file, for all the other files too. Everything in one single file.

Of course I have searched SO and found some methods. But none is working completely.

First try:

Adding sys.stdout = open(self.logfilename, 'w') to my constructor. This makes line 1 and 2 of the above showed output be written to my log file. The rest is still shown on console.

Second try:

Additionaly, in analyseCpps I use:

call(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout)

This makes my log file to be:

Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...


********************************************************************** 
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc

and the console output is:

[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.

Not what I want.

Third try:

Using Popen with pipe. sys.stdout is back to default.

As preliminary work analyseCpps now is:

for sourcefile in self.files['.cc'] + self.files['.cpp']:

    print '\n'*2, '*'*70, '\n', sourcefile
    p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=subprocess.PIPE)
    p.stdout.read()

p.stdout.read() shows only the last line of my desired output (line 7 in code box 3)

Fourth try:

Using subprocess.Popen(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=open(self.logfilename, 'a+')) just writes the one line Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc... to my logfile, the rest is shown on the console.

Fifth try:

Instead of subprocess.Popen I use os.system, so my calling command is:

os.system('C:\CodeAnalysis\cppcheck\cppcheck --enable=all %s >> %s' % (sourcefile, self.logfilename))

This results in the same log file as my fourth try. If I type the same command directly in the windows console the result is the same. So I guess it it is not exactly a python problem but still:

If it is on the console there must be a way to put it in a file. Any ideas?

E D I T

Foolish me. I'm still a noob so I forgot about the stderr. That's where the decisive messages are going to.

So now I have:

def analyseCpps(self):

    for sourcefile in self.files['.cc'] + self.files['.cpp']:
        p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stderr=subprocess.PIPE)
        with open(self.logfilename, 'a+') as logfile:
            logfile.write('%s\n%s\n' % ('*'*70, sourcefile))
            for line in p.stderr.readlines():
                logfile.write('%s\n' % line.strip())

and it's working fine.

ANOTHER EDIT

according to Didier's answer:

with sys.stdout = open(self.logfilename, 'w', 0) in my constructor:

def analyseCpps(self):

    for sourcefile in self.files['.cc'] + self.files['.cpp']:
        print '\n'*2, '*'*70, '\n', sourcefile
        p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout, stderr=sys.stdout)

Thanks for your fast comments and answers, you are all right. — seesharp, Aug 12 '15 at 12:14
related: [Redirect stdout to a file in Python?](http://stackoverflow.com/q/4675728/4279) — jfs, Aug 13 '15 at 16:28
unrelated: do not use `for line in p.stderr.readlines():`, just `for line in p.stderr:` works. Also, you could write `shutil.copyfileobj(p.stderr, logfile)`. Also, you could use `stderr=subprocess.STDOUT`. Also, you could [use my answer to redirect stdout at file descriptor level and merge stderr into stdout](http://stackoverflow.com/a/22434262/4279) -- drop `stdout`, `stderr` parameters in this case -- they are handled automatically. — jfs, Aug 13 '15 at 16:34

score 2 · Accepted Answer · answered Aug 12 '15 at 12:08

2

There are several problems:

you should redirect both stdout and stderr
you should use unbuffered files if you want to mix normal print and the output of launched commands.

Something like this:

import sys, subprocess

# Note the 0 here (unbuffered file)
sys.stdout = open("mylog","w",0)

print "Hello"
print "-----"

subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"

print "End"

answered Aug 12 '15 at 12:08

Didier Spezia

70,911
12
189
154

This is actually pretty cool. I will re-change my implementation. – seesharp Aug 12 '15 at 12:19
@seesharp: it seems fragile to redirect output of every and each subprocess call manually and it fails if some code writes to stdout directly (`os.write(1, b"won't catch me")`). A better alternative is to redirect the output in the code that runs the Python script: `python your_script.py &> log` (bash) or redirect using `os.dup2()` (works on Windows too). See the links in my comments to your question. – jfs Aug 13 '15 at 16:40
I use Python 2.7 (didn't mention, shame on me) so there is no `contextlib.redirect_stdout()`. Furthermore I decided not to redirect **every** output to a file. As the whole script takes a decent amount of time to run I use print statements to show progress and the subprocesses' outputs are directed directly to my logfile. So far it works, so I will leave it this way but I keep your answer in mind for the future. – seesharp Aug 14 '15 at 06:35

score 1 · Answer 2 · answered Aug 12 '15 at 12:05

You need to redirect stderr too, you can use STDOUT or pass the file object to stderr=:

from subprocess import check_call,STDOUT
with open("log.txt","w") as f:
     for sourcefile in self.files['.cc'] + self.files['.cpp']:
        check_call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile],
                   stdout=f, stderr=STDOUT)

score 1 · Answer 3 · answered Aug 12 '15 at 12:06

Try to redirect stdout and stderr to a logfile:

import subprocess

def analyseCpps(self):
     with open("logfile.txt", "w") as logfile:
         for sourcefile in self.files['.cc'] + self.files['.cpp']:
             print '\n'*2, '*'*70, '\n', sourcefile
             call(['C:\\CodeAnalysis\\cppcheck\\cppcheck',
                   '--enable=all', sourcefile], stdout=logfile,
                   stderr=subprocess.STDOUT)

In this example the filename is hardcoded, but you should be able to change that easily (to your self.logfilename or similar).

Well I just realized, the messages go to the stderr and was about to type an answer myself :-). So you are right. I will edit the working code in the answer. — seesharp, Aug 12 '15 at 12:09

python script using subprocess, redirect ALL output to file

3 Answers3