How to monitor Python files for changes?

Question

I want to restart my Python web application, if code gets changed. But there could be a large number of files that could be changed, since files in imported modules could change ...

How to get the actual file names from imported packages / modules?

How can modified Python files be detected efficiently? Is there a library to do that?

Answer 1

Shameless plug. There's also http://github.com/gorakhargosh/watchdog that I'm working on to do exactly this.

HTH.

Answer 2

Here's an example of how this could be implemented using pyinotify (ie., on Linux).

from importlib import import_module

class RestartingLauncher:

    def __init__(self, module_name, start_function, stop_function, path="."):
        self._module_name = module_name
        self._filename = '%s.py' % module_name
        self._start_function = start_function
        self._stop_function = stop_function
        self._path = path
        self._setup()

    def _setup(self):
        import pyinotify
        self._wm = pyinotify.WatchManager()

        self._notifier = pyinotify.ThreadedNotifier(
                self._wm, self._on_file_modified)
        self._notifier.start()

        # We monitor the directory (instead of just the file) because
        # otherwise inotify gets confused by editors such a Vim.
        flags = pyinotify.EventsCodes.OP_FLAGS['IN_MODIFY']
        wdd = self._wm.add_watch(self._path, flags)

    def _on_file_modified(self, event):
        if event.name == self._filename:
            print "File modification detected. Restarting application..."
            self._reload_request = True
            getattr(self._module, self._stop_function)()

    def run(self):
        self._module = import_module(self._module_name)

        self._reload_request = True
        while self._reload_request:
            self._reload_request = False
            reload(self._module)
            getattr(self._module, self._start_function)()

        print 'Bye!'
        self._notifier.stop()

def launch_app(module_name, start_func, stop_func):
    try:
        import pyinotify
    except ImportError:
        print 'Pyinotify not found. Launching app anyway...'
        m = import_module(self._module_name)
        getattr(m, start_func)()
    else:
        RestartingLauncher(module_name, start_func, stop_func).run()

if __name__ == '__main__':
    launch_app('example', 'main', 'force_exit')

The parameters in the launch_app call are the filename (without the ".py"), the function to start execution and a function that somehow stops the execution.

Here's a stupid example of an "app" that could be (re-)launched using the previous code:

run = True

def main():
    print 'in...'
    while run: pass
    print 'out'

def force_exit():
    global run
    run = False

In a typical application where you'd want to use this, you'd probably have a main loop of some sort. Here's a more real example, for a GLib/GTK+ based application:

from gi.repository import GLib

GLib.threads_init()
loop = GLib.MainLoop()

def main():
    print "running..."
    loop.run()

def force_exit():
    print "stopping..."
    loop.quit()

The same concept works for most other loops (Clutter, Qt, etc).

Monitoring several code files (ie. all files that are part of the application) and error resilience (eg. printing exceptions and waiting in an idle loop until the code is fixed, then launching it again) are left as exercises for the reader :).

Note: All code in this answer is released under the ISC License (in addition to Creative Commons).

Answer 3

gamin is another option which is slightly less Linux-specific.

Answer 4

I'm not sure how you would implement the 'reload application' operation in your circumstance; reloading a changed module with the reload built-in probably won't cut it.

But as far as detecting whether or not there was a change, the following would be one way to approach it.

• Most python modules have a __file__ attribute.
• All loaded modules are stored in sys.modules.
• We can walk through sys.modules at some interval, and look for changes on disk for each module in turn

Sometimes __file__ points to a .pyc file instead of a .py file, so you might have to chop off the trailing c. Sometimes a .pyc file exists but a .py doesn't exist; in a robust system you'd have to allow for this.

A proof of concept of code to do this (not robust):

_module_timestamps = {}
_checking = False

def run_checker():
    global _checking
    _checking = True
    while _checking:
        for name, module in sys.modules.iteritems():
            if hasattr(module, '__file__'):
                filename = module.__file__
                if filename.endswith('.pyc'):
                    filename = filename[:-1]
                mtime = os.stat(filename).st_mtime
                if name not in _module_timestamps:
                    _module_timestamps[name] = mtime
                else:
                    if mtime > _module_timestamps[name]:
                        do_reload(name)
            else:
                'module %r has no file attribute' % (name,)
        time.sleep(1)

def do_reload(modname):
    print 'I would reload now, because of %r' % (modname,)

check_thread = threading.Thread(target=run_checker)
check_thread.daemon = True
check_thread.start()

try:
    while 1:
        time.sleep(0.1)
except KeyboardInterrupt:
    print '\nexiting...'

Answer 5

This is operating system specific. For Linux, there is inotify, see e.g. http://github.com/rvoicilas/inotify-tools/