How to get subprocess output during its running?

Question

I've got 2 files main.py and infinit.py, like below:

main.py

#!/usr/bin/python

import logging
import subprocess
import sys

logging.basicConfig(level=logging.INFO)


def forever():
    cmd = [sys.executable, 'infinit.py']
    while 1:
        try:
            print 'running new instance of:'
            print ' '.join(cmd)
            popen = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                stderr=subprocess.PIPE, universal_newlines=True)

            for line in iter(popen.stderr.readline, ""):
                print line,
            for line in iter(popen.stdout.readline, ""):
                print line,

        except Exception as e:
            print 'Something bad happend'
            logging.error(e)


if __name__ == '__main__':
    forever()

infinit.py

#!/usr/bin/python

import logging

logging.basicConfig(level=logging.INFO)

i = 0
while 1:
    i += 1
    logging.info('i: {0}'.format(i))
    print i

I run main.py and I want to see both (printing and logging ) in my console. I also want it to run on windows and linux. Additionally is it possible that it works (printing and logging) in windows Idle?

I had a problem similar to this, this is how I solved it. http://stackoverflow.com/questions/9137010/asynchronously-retrieving-information-from-a-process/9138042#9138042. — John, Oct 29 '12 at 16:38

score 0 · Answer 1 · answered Oct 29 '12 at 16:45

Add this lines at the begging of your program, and put the same LOG_FILENAME, then you can use tail -f unix command to see the output.

import logging
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG, filemode = 'w+')
logging.getLogger().setLevel(logging.DEBUG)

score 0 · Accepted Answer · edited May 23 '17 at 10:34

I found two solutions, both should works on Linux and Windows and even in Windows Idle:

The simplier:

In the parent process print the child's stderr:

#!/usr/bin/python

import logging
import subprocess
import sys

def forever():
    cmd = [sys.executable, 'infinit.py']
    while 1:
        try:
            print 'running new instance of:'
            print ' '.join(cmd)
            popen = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                stderr=subprocess.PIPE, universal_newlines=True)

            for line in iter(popen.stderr.readline, ""):
                print line,

        except Exception as e:
            print 'Something bad happend'
            logging.error(e)

if __name__ == '__main__':
    forever()

For now:

Parent prints children logging (because logging by default send messages to stderr adn stderr we are printing)
Parent not print children printings, so:

infinit.py

#!/usr/bin/python

import logging
import sys
import time

logging.basicConfig(level=logging.INFO)

sys.stdout = sys.stderr

i = 0
while 1:
    i += 1
    logging.info('i: {0}'.format(i))
    print i
    time.sleep(0.2)

Another approch was here:

Intercepting stdout of a subprocess while it is running

And my code adaptated to the solution

main.py

#!/usr/bin/python

import logging
import subprocess
import sys

def forever():
    CMD = [sys.executable, 'infinit.py']
    while 1:
        try:
            print 'running new instance of:'
            print ' '.join(CMD)

            popen = subprocess.Popen(CMD, stdout=subprocess.PIPE)
            for line in iter(popen.stdout.readline, ""):
                print line,

        except Exception as e:
            print 'Something bad happend'
            logging.error(e)
        finally:
            print


if __name__ == '__main__':
    forever()

inifinit.py

#!/usr/bin/python

import logging
import sys
import time


class FlushFile(object):
    """Write-only flushing wrapper for file-type objects."""
    def __init__(self, f):
        self.f = f
    def write(self, x):
        self.f.write(x)
        self.f.flush()

logging.basicConfig(level=logging.INFO)

sys.stdout = FlushFile(sys.stdout)


i = 0
while 1:
    i += 1
    logging.info('i: {0}'.format(i))
    print i
    time.sleep(0.2)

How to get subprocess output during its running?

2 Answers2