Tail file graphically with PyQt

Question

I'm trying to do what sounds fairly simple but I keep running in to all sorts of problems. I'm trying to create a GUI that can tail several files at the same time using PyQt. I saw this answer on how to tail a file in pure Python

How can I tail a log file in Python?

I have tried using this code inside of a QThread. The issues I'm having here are that the tail process never stops by itself; it needs to be killed. It should be killed when the GUI is closed. The other issues I'm getting with this specific solution below is

QThread: Destroyed while thread is still running

and

QWaitCondition::wakeAll(): mutex lock failure: 

and

QThread: Destroyed while thread is still running
Traceback (most recent call last):
  File "./tailer.py", line 27, in run
    self.emit(SIGNAL('newline'), line.rstrip())
RuntimeError: underlying C/C++ object has been deleted

Other implementations I've tried have had the tail process complaining about a broken pipe but those stopped appearing once I did stderr=PIPE as well. I'm worried now that I could be missing errors since I never read from stderr (since it would block and there shouldn't be any output).

To get the errors fire this up trying to tail 3 different file. I wrote another script that loops and writes to those 3 files doing a sleep of 0.1 seconds. I close the GUI and start it up over and over again. Sometimes I get errors sometimes I don't.

Please tell me what I'm doing wrong here.

#!/usr/bin/env python

from PyQt4.QtCore import *
from PyQt4.QtGui import *

import os
from subprocess import Popen, PIPE

class Tailer(QThread):

    def __init__(self, fname, parent=None):
        super(Tailer, self).__init__(parent)
        self.fname = fname
        self.connect(self, SIGNAL('finished()'), self.cleanup)

    def cleanup(self):
        print 'CLEANING UP'
        self.p.kill()
        print 'killed'

    def run(self):
        command = ["tail", "-f", self.fname]
        print command
        self.p = Popen(command, stdout=PIPE, stderr=PIPE)
        while True:
            line = self.p.stdout.readline()
            self.emit(SIGNAL('newline'), line.rstrip())
            if not line:
                print 'BREAKING'
                break

    def foo(self):
        self.p.kill()

class TailWidget(QWidget):
    def __init__(self, fnames, parent=None):
        super(TailWidget, self).__init__(parent)
        layout = QGridLayout()
        self.threads = {}
        self.browsers = {}
        for i, fname in enumerate(fnames):
            if not os.path.exists(fname):
                print fname, "doesn't exist; creating"
                p = Popen(['touch', fname], stdout=PIPE, stderr=PIPE)
                out, err = p.communicate()
                ret = p.wait()
                assert ret == 0
            t = Tailer(fname, self)
            self.threads[fname] = t
            b = QTextBrowser()
            self.browsers[fname] = b
            layout.addWidget(QLabel('Tail on %s' % fname), 0, i)
            layout.addWidget(b, 1, i)
            self.connect(t, SIGNAL("newline"), b.append)
            t.start()
        self.setLayout(layout)

    def closeEvent(self, event):
        for fname, t in self.threads.items():
            t.foo()

if __name__ == '__main__':
    import sys
    app = QApplication(sys.argv)
    tw = TailWidget(sys.argv[1:])
    tw.show()
    sys.exit(app.exec_())

Answer 1

The problem is that the main thread isn't waiting on the background threads.

You tell them to stop here:

def closeEvent(self, event):
    for fname, t in self.threads.items():
        t.foo()

So, this kills all of the subprocesses, which will make all of the background threads quit eventually. But it won't make them stop immediately. That won't happen until the next time each one gets to its readline.

After killing the subprocesses, you return, letting Qt immediately close your window and destroy your widget. Any background threads that try to send a signal to that widget will then fail.

Imagine that thread 1 has done a readline, and is in the middle of its rstrip when thread 0 tries to close down. So, thread 0 kills thread 1's subprocess, then deletes the main widget. Thread 1 finishes its rstrip and calls emit, and it's now sending to a deleted widget.