How to seamlessly wrap Bash shell IO in Python

Question

How do you wrap a bash shell session in a Python script so that Python can store the stdout and stderr to a database, and occasionally write to stdin?

I tried using subprocess with a tee-like Python class to redirect the IO, but it seems to use fileno to bypass Python entirely.

shell.py:

import os
import sys
from StringIO import StringIO
from subprocess import Popen, PIPE

class TeeFile(StringIO):
    def __init__(self, file, auto_flush=False):
        #super(TeeFile, self).__init__()
        StringIO.__init__(self)
        self.file = file
        self.auto_flush = auto_flush
        self.length = 0

    def write(self, s):
        print 'writing' # This is never called!!!
        self.length += len(s)
        self.file.write(s)
        #super(TeeFile, self).write(s)
        StringIO.write(self, s)
        if self.auto_flush:
            self.file.flush()

    def flush(self):
        self.file.flush()
        StringIO.flush(self)

    def fileno(self):
        return self.file.fileno()

cmd = ' '.join(sys.argv[1:])
stderr = TeeFile(sys.stderr, True)
stdout = TeeFile(sys.stdout, True)

p = Popen(cmd, shell=True, stdin=PIPE, stdout=stdout, stderr=stderr, close_fds=True)

e.g. Running python shell.py ping google.com runs the correct command and shows output, but Python never sees the stdout.

Have a look at this: http://stackoverflow.com/questions/616645/how-do-i-duplicate-sys-stdout-to-a-log-file-in-python and also this pure python answer to the same question: http://stackoverflow.com/questions/616645/how-do-i-duplicate-sys-stdout-to-a-log-file-in-python/3423392#3423392 — snies, Jul 02 '12 at 16:34
@snies, Thanks. That looks like that might only cover half of my question (the stdout and not stdin), but I'll check it out. — Cerin, Jul 02 '12 at 17:39
@snies, Looking at it further, it seems that most of those answers just handle logging stdout across an arbitrary number of subprocesses, and most of the solutions seem to have the same error as my example code. i.e. the Python write() method is never going to be called. — Cerin, Jul 02 '12 at 17:50
Are you familiar with the [script](http://linux.die.net/man/1/script) utility? — Dennis Williamson, Jul 02 '12 at 19:20
@DennisWilliamson, Yes. I don't see how that's relevant. I don't believe script can record the session to a database, nor insert text into the session based on arbitrary logic... — Cerin, Jul 02 '12 at 20:06

score 1 · Answer 1 · answered Jul 03 '12 at 21:11

#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.internet import protocol
from twisted.internet import reactor
import re

class MyPP(protocol.ProcessProtocol):
    def __init__(self, verses):
        self.verses = verses
        self.data = ""
    def connectionMade(self):
        print "connectionMade!"
        for i in range(self.verses):
            self.transport.write("Aleph-null bottles of beer on the wall,\n" +
                                 "Aleph-null bottles of beer,\n" +
                                 "Take one down and pass it around,\n" +
                                 "Aleph-null bottles of beer on the wall.\n")
        self.transport.closeStdin() # tell them we're done
    def outReceived(self, data):
        print "outReceived! with %d bytes!" % len(data)
        self.data = self.data + data
    def errReceived(self, data):
        print "errReceived! with %d bytes!" % len(data)
    def inConnectionLost(self):
        print "inConnectionLost! stdin is closed! (we probably did it)"
    def outConnectionLost(self):
        print "outConnectionLost! The child closed their stdout!"
        # now is the time to examine what they wrote
        #print "I saw them write:", self.data
        (dummy, lines, words, chars, file) = re.split(r'\s+', self.data)
        print "I saw %s lines" % lines
    def errConnectionLost(self):
        print "errConnectionLost! The child closed their stderr."
    def processExited(self, reason):
        print "processExited, status %d" % (reason.value.exitCode,)
    def processEnded(self, reason):
        print "processEnded, status %d" % (reason.value.exitCode,)
        print "quitting"
        reactor.stop()

pp = MyPP(10)
reactor.spawnProcess(pp, "wc", ["wc"], {})
reactor.run()

Thats the Twisted way to handle command IO as a protocol. Btw you too complicate your script with StringIO. Rather check Popen.communicate() method. Note that stdin/stdout are file descriptors and need to be read in parallel cause their buffers will overflow if longer output. If you want to stream huge data through this, rather use that Twisted way, or in case Popen way fire separate thread for reading stdout may be by lines and putting them to DB immediately. Twisted howto on process protocol.

How to seamlessly wrap Bash shell IO in Python

1 Answers1