Python Watchdog process existing files on startup

Question

I have a simple Watchdog and Queue process to monitor files in a directory. Code taken from https://camcairns.github.io/python/2017/09/06/python_watchdog_jobs_queue.html

import time
from watchdog.events import PatternMatchingEventHandler
from watchdog.observers import Observer
from queue import Queue
from threading import Thread

dir_path = "/data"

def process_queue(q):

    while True:
        if not q.empty():
            event = q.get()
            print("New event %s" % event)

        time.sleep(5)


class FileWatchdog(PatternMatchingEventHandler):

    def __init__(self, queue, patterns):
        PatternMatchingEventHandler.__init__(self, patterns=patterns)
        self.queue = queue

    def process(self, event):
        self.queue.put(event)

    def on_created(self, event):
        self.process(event)


if __name__ == '__main__':

    watchdog_queue = Queue()

    worker = Thread(target=process_queue, args=(watchdog_queue,))
    worker.setDaemon(True)
    worker.start()

    event_handler = FileWatchdog(watchdog_queue, patterns="*.ini")
    observer = Observer()
    observer.schedule(event_handler, path=dir_path)
    observer.start()

    try:
        while True:
            time.sleep(2)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

Once the process is running new files are processed correctly. However if I restart the process and a file already exists in the directory it is ignored.

I have tried to create a dict to add to the queue

    for file in os.listdir(dir_path):
        if file.endswith(".ini"):
             file_path = os.path.join(dir_path, file)
             event = {'event_type' : 'on_created', 'is_directory' : 'False', 'src_path' : file_path}
             watchdog_queue.put(event)

but it's expecting an object of type (class 'watchdog.events.FileCreatedEvent') and I can't work out how to create this.

Alternatively I can see in the Watchdog documentation (class watchdog.utils.dirsnapshot.DirectorySnapshot) but I cannot work out how to run this and add it to the queue.

Any suggestions on how I can add existing files to the queue on startup ?

score 5 · Answer 1 · edited Apr 14 '20 at 16:29

5

This code should do what you are trying to achieve.

from watchdog.events import FileCreatedEvent

# Loop to get all files; dir_path is your lookup folder.

for file in os.listdir(dir_path):
    filename = os.path.join(dir_path, file)
    event = FileCreatedEvent(filename)
    watchdog_queue.put(event)

edited Apr 14 '20 at 16:29

apaderno

28,547
16
75
90

answered Apr 14 '20 at 12:13

Prem

51
1
3

1

If you want to exclude subdirectories, and `if os.path.isfile(filename):` under the `filename =` line – Tyler Dane Apr 20 '20 at 20:43
Hello @apaderno, if I have ```class Handler(FileSystemEventHandler): def on_any_event(event): filename=event.src_path; if event.event_type in ['created']: callpython(filename)```, May I know where should I add the for loop in the script? – Susan Jan 18 '23 at 14:49
@Susan You should ask that to Prem, who wrote this answer. – apaderno Jan 18 '23 at 17:16

score 0 · Answer 2 · answered Oct 20 '21 at 06:58

I stumbled over the same problem and maybe this solution is for you too. At least on linux this works like a charm.

Add the "on_modified" method

class FileWatchdog(PatternMatchingEventHandler):

def __init__(self, queue, patterns):
    PatternMatchingEventHandler.__init__(self, patterns=patterns)
    self.queue = queue

...

def on_modified(self, event):
    self.process(event)

Now after starting the observer, loop through all files in directory and "touch" them, so they will be "modified".

# Loop to get all files; dir_path is your lookup folder.

for file in os.listdir(dir_path):
    filename = os.path.join(dir_path, file)
    os.popen(f'touch {filename}')

No need to add special filters as your FileHandler will handle that.

Python Watchdog process existing files on startup

2 Answers2

Linked