Logging to multiple log files from different classes in Python

Question

I want to write a Python class which uses Python logging. This Python class will be responsible for the creating a file with a given name in init function.

I want to create a object of the above class in two or more classes and expect two or files getting generated.

I tried writing this class but I am not able to create multiple files.

Can anyone guide me how do I do that?

I have created the following class:

class Logger:
def __init__(self, log_filename = "test.log"):
    if not os.path.exists("LogFiles"):
        os.makedirs("LogFiles")
    self.Logger = logging.getLogger("main")
    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s : %(message)s',
                        filename= log_filename,
                        filemode='w')           # change filemode to 'w' to overwrite file on each run

    consoleHandler = logging.StreamHandler()
    consoleHandler.setLevel(logging.DEBUG)
    formatter = logging.Formatter('%(asctime)s - %(message)s')
    consoleHandler.setFormatter(formatter)
    logging.getLogger('').addHandler(consoleHandler)      # Add to the root logger
    self.Logger.info("Starting new logging sessions")


def writeToFile(self, line):
    if self.Logger.propagate == True:
        self.Logger.debug(line)

def closeFile(self):

    if self.Logger.propagate == True:
        self.Logger.propagate = False

Answer 1

Sounds like the internals of your class should probably have a Logger and that you'll want to add a FileHandler to the Logger. You might want to consider just using a factory method that creates Loggers and adds the handler instead of implementing your own class. You may need to create the directories that hold the log files. See this answer for advice on creating directories.

Edit:

I don't think you need to write your own Logger class. Python's logging module has all the pieces you need. You probably just need a factory method. The key to realize is you need to create two separate, completely independent logging objects. You do this with logging.getLogger, and any time you pass it a different name, it gives you a different logger. You can use anything you want for the logger's name. For sure, you want to stay away from basicConfig for what you're doing. It's designed to be something simple for people who just want one Logger not doing anything too special.

I think this demonstrates the functionality you're after. The key is create two different loggers with different handlers. Then use them separately. Keep in mind that my second call to logging.getLogger doesn't create a new logger; it gets the one we set up initially in setup_logger.

log_test.py:

from __future__ import absolute_import

import logging

def setup_logger(logger_name, log_file, level=logging.INFO):
    l = logging.getLogger(logger_name)
    formatter = logging.Formatter('%(asctime)s : %(message)s')
    fileHandler = logging.FileHandler(log_file, mode='w')
    fileHandler.setFormatter(formatter)
    streamHandler = logging.StreamHandler()
    streamHandler.setFormatter(formatter)

    l.setLevel(level)
    l.addHandler(fileHandler)
    l.addHandler(streamHandler)    

def main():
    setup_logger('log1', r'C:\temp\log1.log')
    setup_logger('log2', r'C:\temp\log2.log')
    log1 = logging.getLogger('log1')
    log2 = logging.getLogger('log2')

    log1.info('Info for log 1!')
    log2.info('Info for log 2!')
    log1.error('Oh, no! Something went wrong!')

if '__main__' == __name__:
    main()

Sample run:

C:\temp>C:\Python\27\python.exe logtest.py
2013-06-12 02:00:13,832 : Info for log 1!
2013-06-12 02:00:13,832 : Info for log 2!
2013-06-12 02:00:13,832 : Oh, no! Something went wrong!

log1.log:

2013-06-12 02:00:13,832 : Info for log 1!
2013-06-12 02:00:13,832 : Oh, no! Something went wrong!

log2.log:

2013-06-12 02:00:13,832 : Info for log 2!

Answer 2

This is better handled using dictConfig. You can then specify logging to two seperate files. I utilize the second_logger only when an API post is made so that I can log external data to the second log.

import os, logging
from logging.config import dictConfig

FORMAT = "%(asctime)s {app} [%(thread)d] %(levelname)-5s %(name)s - %(message)s. [file=%(filename)s:%(lineno)d]"
DATE_FORMAT = None


def setup_logging(name, level="INFO", fmt=FORMAT):
    formatted = fmt.format(app=name)
    log_dir = r'C:/log_directory'
    if not os.path.exists(log_dir):
        os.makedirs(log_dir)

    logging_config = {
        "version": 1,
        'disable_existing_loggers': False,
        "formatters": {
            'standard': {
                'format': formatted
            }
        },
        "handlers": {
            'default': {
                'class': 'logging.StreamHandler',
                'formatter': 'standard',
                'level': level,
                'stream': 'ext://sys.stdout'
            },
            'file': {
                'class': 'logging.handlers.TimedRotatingFileHandler',
                'when': 'midnight',
                'utc': True,
                'backupCount': 5,
                'level': level,
                'filename': '{}/app_manager.log'.format(log_dir),
                'formatter': 'standard',
            },
            'file2': {
                'class': 'logging.handlers.TimedRotatingFileHandler',
                'when': 'midnight',
                'utc': True,
                'backupCount': 5,
                'level': level,
                'filename': '{}/unified_log.log'.format(log_dir),
                'formatter': 'standard',
            }
        },
        "loggers": {
            "": {
                'handlers': ['default', 'file'],
                'level': level
            },
            "second_log": {
                'handlers': ['default', 'file2'],
                'level': level
            }
        }
    }

    dictConfig(logging_config)

log.setup_logging(name="log-name", level=LEVEL
logger = logging.getLogger(__name__)
second_logger = logging.getLogger('second_log')
second_logger.propagate = False