python-dynamically change log's directory when log content is updated

Question

I'd like to change the log's directory depending on the time (specifically the hour) when new log is created.

For example, let's say the message aaa is saved in a log file in directory d:\log\191105\09\log.log at 09:59 during running a program.

As the time passes, new log bbb is saved in a log file but the directory should be different, d:\log\191105\10.log at 10:00 without terminating the program.

My log manager code is following,

import logging
import datetime
import psutil
import os, os.path

class LogManager:
    today = datetime.datetime.now()
    process_name = psutil.Process().name()
    process_id = str(psutil.Process().pid)
    log_dir = "D:/LOG/" + today.strftime("%Y%m%d") + "/" + today.strftime("%H") + "/"
    log_filename = process_name + '_[' + process_id + ']_' + today.strftime("%Y-%m-%d_%H") + '.log'
    if not os.path.exists(log_dir):
        os.makedirs(log_dir)

    # date/directory config
    log = logging.getLogger('mypython')
    log.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s||%(levelname)s||%(message)s')
    fileHandler = logging.FileHandler(log_dir+log_filename)
    fileHandler.setFormatter(formatter)

    log.addHandler(fileHandler)

    def update(self):
        self.today = datetime.datetime.now()
        old_log_dir = self.log_dir
        self.log_dir = "D:/LOG/" + self.today.strftime("%Y%m%d") + "/" + self.today.strftime("%H") + "/"

        if (old_log_dir == self.log_dir):
            return

        self.log_filename = self.process_name + '_[' + self.process_id + ']_' + self.today.strftime("%Y-%m-%d_%H") + '.log'
        if not os.path.exists(self.log_dir):
            os.makedirs(self.log_dir)

        self.log = logging.getLogger('mypython')

        # here, i wanna update my filehandler
        for hdlr in self.log.handlers[:]:
            if isinstance(hdlr, self.log.FileHander):
                self.log.removeHandler(hdlr)

        self.fileHandler = logging.FileHandler(self.log_dir+self.log_filename)
        self.fileHandler.setFormatter(self.formatter)

        self.log.addHandler(self.fileHandler)

    def i(self, ex):
        self.update()
        self.log.info(ex)

    def w(self, ex):
        self.update()
        self.log.warning(ex)

and I call these functions like

import LogManager as lm

logManager = lm.LogManager()
logManager.i('message')

It works well, but seems like it does not update its fileHandler after passing the hour.

I tried to find if the log has updateHandler method... but it doesn't .

What should I do??

Answer 1

You can do this much easier by using what is already in the logging library. Specifically there is a TimedRotatingFileHandler for which you only need to change how it names the files it creates. I've created a working minimal demo for you which changes the folder that logs get saved into on every second. Edit: change to rollover on every full hour instead of every second as per comment below

import os
import logging
from time import sleep, strftime
from logging.handlers import TimedRotatingFileHandler

def namer(default_name):
    head, tail = os.path.split(default_name)
    folder_name = 'logs'+strftime('%H%M%S') 
    folder = os.path.join(head, folder_name)
    try:
        os.mkdir(folder)
    except:
        # folder already exists
        pass
    return os.path.join(folder, tail)

logger = logging.getLogger()
handler = TimedRotatingFileHandler('base.log', when='H')
handler.namer = namer
# set the rollover time to the next full hour
handler.rolloverAt = datetime.now().replace(microsecond=0,second=0,minute=0).timestamp() + 60*60
logger.addHandler(handler)

logger.warning('test1')
sleep(2)
logger.warning('test2')

Answer 2

The problem with @blues solution is that it assumes namer names the current log, which is incorrect and will lead to logging issues.

The intention of the BaseRotatingHandler logic as shown here: https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing is to customise the logic of what happens to the rotated files. The rotation logic assumes the current log file name is fixed (hence why I have fixed the path to use the path: logs/today/my_app.log

Below is a full working example that keeps the current logs in a 'today' directory. It then rotates based on the 'when' parameter given to the TimedRotatingFileHandler. On rotation it creates a directory with the previous interval's date time stamp in format %Y%m%d%M. It then rotates everything in 'today' in to that folder.

On the configured rotation time it will then rotate the logs from the past interval in to a directory with name given by the timestamp of now - interval seconds.

For your specific example you would change when to be when='H' and the date_str time format in namer() to be:

date_str = f"{ts.strftime('%Y%m%d')}/{ts.strftime('%H')}"

import datetime
import logging
import time
import logging.handlers as handlers
from pathlib import Path
from typing import List


class DateFormattedPathRotator(handlers.TimedRotatingFileHandler):

    def rotate(self, source: str, dest: str) -> None:
        """
        :param source: The current name of the log file
        :param dest: The name the log file will be rotated to (given by the return of namer)
        :return:
        """
        # rotate existing logs to a folder for the last minute
        print(f"rotating from {source} to {dest}")
        rotation_source = Path(source)
        rotation_destination = Path(dest)
        rotation_destination.parent.mkdir(parents=True, exist_ok=True)
        rotation_source.replace(rotation_destination)

    def namer(self, name):
        """
        :param name: The current name of the log file
        :return: The destination name of the rotated log file
        """
        # the interval is set in the parent class depending on 'when'
        # we will rotate the logs produced within the last 'interval' seconds in to their own time formatted directory
        time_stamp = datetime.datetime.utcnow() - datetime.timedelta(seconds=self.interval)
        date_str = time_stamp.strftime("%Y%m%d%M")
        rotation_path = Path(f"logs/{date_str}")
        rotation_destination = rotation_path / Path(name).name
        return str(rotation_destination)


if __name__ == "__main__":
    log_path_str = f"logs/today/my_app.log"
    log_path = Path(log_path_str)
    log_path.parent.mkdir(parents=True, exist_ok=True)
    file_handler = DateFormattedPathRotator(
        log_path,
        when='m',
        interval=1,
    )
    log_handlers: List[logging.Handler] = [logging.StreamHandler(), file_handler]
    logging.basicConfig(
        level="INFO",
        format="[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s",
        handlers=log_handlers,
    )
    logger = logging.getLogger(__name__)
    i = 0
    while True:
        logger.info(f"hello world {i}")
        time.sleep(10)
        i += 1

Running the example as is gives the following output to console:

[2023-07-13 15:42:10,092] [INFO] [__main__]: hello world 0
[2023-07-13 15:42:20,102] [INFO] [__main__]: hello world 1
[2023-07-13 15:42:30,112] [INFO] [__main__]: hello world 2
[2023-07-13 15:42:40,123] [INFO] [__main__]: hello world 3
[2023-07-13 15:42:50,133] [INFO] [__main__]: hello world 4
[2023-07-13 15:43:00,143] [INFO] [__main__]: hello world 5
[2023-07-13 15:43:10,153] [INFO] [__main__]: hello world 6
rotating from /tmp/pycharm_project_89/logs/today/my_app.log to logs/2023071342/my_app.log.2023-07-13_15-42
[2023-07-13 15:43:20,164] [INFO] [__main__]: hello world 7
[2023-07-13 15:43:30,174] [INFO] [__main__]: hello world 8
[2023-07-13 15:43:40,184] [INFO] [__main__]: hello world 9
[2023-07-13 15:43:50,194] [INFO] [__main__]: hello world 10
[2023-07-13 15:44:00,204] [INFO] [__main__]: hello world 11
[2023-07-13 15:44:10,214] [INFO] [__main__]: hello world 12
rotating from /tmp/pycharm_project_89/logs/today/my_app.log to logs/2023071343/my_app.log.2023-07-13_15-43
[2023-07-13 15:44:20,225] [INFO] [__main__]: hello world 13
[2023-07-13 15:44:30,235] [INFO] [__main__]: hello world 14

and creates the following log structure:

tree /tmp/pycharm_project_89/logs/
/tmp/pycharm_project_89/logs/
├── 2023071342
│   └── my_app.log.2023-07-13_15-42
├── 2023071343
│   └── my_app.log.2023-07-13_15-43
└── today
    └── my_app.log

3 directories, 3 files

with log file content:

cat /tmp/pycharm_project_89/logs/2023071342/my_app.log.2023-07-13_15-42
[2023-07-13 15:42:10,092] [INFO] [__main__]: hello world 0
[2023-07-13 15:42:20,102] [INFO] [__main__]: hello world 1
[2023-07-13 15:42:30,112] [INFO] [__main__]: hello world 2
[2023-07-13 15:42:40,123] [INFO] [__main__]: hello world 3
[2023-07-13 15:42:50,133] [INFO] [__main__]: hello world 4
[2023-07-13 15:43:00,143] [INFO] [__main__]: hello world 5

cat /tmp/pycharm_project_89/logs/today/my_app.log
[2023-07-13 15:44:10,214] [INFO] [__main__]: hello world 12
[2023-07-13 15:44:20,225] [INFO] [__main__]: hello world 13
[2023-07-13 15:44:30,235] [INFO] [__main__]: hello world 14