How to change filehandle with Python logging on the fly with different classes and imports

Question

I cannot perform an on-the-fly logging fileHandle change.

For example, I have 3 classes

one.py

import logging
class One():
    def __init__(self,txt="?"):
        logging.debug("Hey, I'm the class One and I say: %s" % txt)

two.py

import logging
class Two():
    def __init__(self,txt="?"):
        logging.debug("Hey, I'm the class Two and I say: %s" % txt)

config.py

import logging
class Config():
    def __init__(self,logfile=None):
        logging.debug("Reading config")
        self.logfile(logfile)

myapp

from one import One
from two import Two
from config import Config
import logging

#Set default logging
logging.basicConfig( 
    level=logging.getLevelName(DEBUG), 
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    filename=None
)

logging.info("Starting with stdout")

o=One(txt="STDOUT")
c=Config(logfile="/tmp/logfile")

# Here must be the code that change the logging configuration and set the filehandler

t=One(txt="This must be on the file, not STDOUT")

If I try loggin.basicConfig() again, it doesn't work.

I think the normal way to do this would be to create a dedicated logger for each class. This is a frequently used pattern. I think the switching and switching back will become problematic. — dsz, May 25 '22 at 07:38

Martijn Pieters · Accepted Answer · 2023-01-12T01:04:19.670

Indeed, logging.basicConfig does nothing if a handler has been set up already:

This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.

You'll need to either add force=True (requires Python 3.8 or newer), or, alternatively, replace the current handler on the root logger:

import logging

fileh = logging.FileHandler('/tmp/logfile', 'a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fileh.setFormatter(formatter)

log = logging.getLogger()  # root logger
for hdlr in log.handlers[:]:  # remove all old handlers
    log.removeHandler(hdlr)
log.addHandler(fileh)      # set the new handler

See the Configuring Logging chapter in the Python Logging HOWTO.

score 22 · Answer 2 · edited Apr 25 '20 at 23:56

The answer provided by @Martijn Pieters works good. However, the code snipper removes all handlers and placed only the file handler back. This will be troublesome if your application has handlers added by other modules.

Hence, the below snippet is designed in such a way to replace only the file handler.

The line if isinstance(hdlr,logging.FileHandler) is the key.

import logging

filehandler = logging.FileHandler('/tmp/logfile', 'a')
formatter = logging.Formatter('%(asctime)-15s::%(levelname)s::%(filename)s::%(funcName)s::%(lineno)d::%(message)s')
filehandler.setFormatter(formatter)
log = logging.getLogger()  # root logger - Good to get it only once.
for hdlr in log.handlers[:]:  # remove the existing file handlers
    if isinstance(hdlr,logging.FileHandler):
        log.removeHandler(hdlr)
log.addHandler(filehandler)      # set the new handler
# set the log level to INFO, DEBUG as the default is ERROR
log.setLevel(logging.DEBUG)

score 12 · Answer 3 · answered Jan 31 '16 at 21:59

I found an easier way than the above 'accepted' answer. If you have a reference to the handler, all you need to do is call the close() method and then set the baseFilename property. When you assign baseFilename, be sure to use os.path.abspath(). There's a comment in the library source that indicates it's needed. I keep my configuration stuff in a global dict() so it's easy to keep the FileHandler reference objects. As you can see below, it only takes 2 lines of code to change a log filename for a handler on the fly.

import logging

def setup_logging():
  global config

  if config['LOGGING_SET']:
    config['LOG_FILE_HDL'].close()
    config['LOG_FILE_HDL'].baseFilename = os.path.abspath(config['LOG_FILE'])

    config['DEBUG_LOG_HDL'].close()
    config['DEBUG_LOG_HDL'].baseFilename = os.path.abspath(config['DEBUG_LOG'])
  else:
    format_str = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
    formatter = logging.Formatter(format_str)

    log = logging.getLogger()

    log.setLevel(logging.DEBUG)

    # add file mode="w" to overwrite
    config['LOG_FILE_HDL'] = logging.FileHandler(config['LOG_FILE'], mode='a')
    config['LOG_FILE_HDL'].setLevel(logging.INFO)
    config['LOG_FILE_HDL'].setFormatter(formatter)
    log.addHandler(config['LOG_FILE_HDL'])

    # the delay=1 should prevent the file from being opened until used.
    config['DEBUG_LOG_HDL'] = logging.FileHandler(config['DEBUG_LOG'], mode='a', delay=1)
    config['DEBUG_LOG_HDL'].setLevel(logging.DEBUG)
    config['DEBUG_LOG_HDL'].setFormatter(formatter)
    log.addHandler(config['DEBUG_LOG_HDL'])

    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(formatter)
    log.addHandler(ch)
    config['LOGGING_SET'] = True

Even if you don't save a reference to the handler, if you only have one handler in your logger you can reference it directly and use this same technique: `mylogger = logging.getLogger('mylogger') # get logger` and then `mylogger.handlers[0].close() mylogger.handlers[0].baseFilename = os.path.abspath('newfilename.log')` — markoan, May 08 '19 at 00:56
@user2179204 you sad "easier" and wrote four times as much lines of code - that is not python way. — khex, Aug 09 '19 at 10:57

T. Haggis · Answer 4 · 2019-01-23T22:17:21.457

I tried to implemented the suggestions on this page from @Martijn Pieters combined with @Arun Thundyill Saseendran. I'm too new to be allowed to comment so I have to post an adjusted answer. In the isinstance call, I had to use 'logging' instead of 'log' to get access to the types (log was an instance) and then the 'FileHander' should be 'FileHandler'. I'm using Python 3.6.

import logging

filehandler = logging.FileHandler('/tmp/logfile', 'a')
formatter = logging.Formatter('%(asctime)-15s::%(levelname)s::%(filename)s::%(funcName)s::%(lineno)d::%(message)s')
filehandler.setFormatter(formatter)
log = logging.getLogger()  # root logger - Good to get it only once.
for hdlr in log.handlers[:]:  # remove the existing file handlers
    if isinstance(hdlr,logging.FileHandler): #fixed two typos here
        log.removeHandler(hdlr)
log.addHandler(filehandler)      # set the new handler
# set the log level to INFO, DEBUG as the default is ERROR
logging.setLevel(log.DEBUG)

Great solution ! I'm using Python flask, and it let me save my messages in flask log. Thanks. — Hilton Fernandes, Jan 28 '22 at 21:04
Arun answer was edited. Only last lines are different. Looks like Victor edited that last line. Do you know which is correct? — pauljohn32, Sep 06 '22 at 22:36

8c6b5df0d16ade6c · Answer 5 · 2022-08-04T11:41:20.143

TL;DR

INDEX = 0

#: change to new file location
logger.handlers[INDEX].setStream( open('/path/to/new/log/file.log', 'a') )

#: change to stdout or stderr
import sys
logger.handlers[INDEX].setStream( sys.stdout ) # or sys.stderr

#: change to another device
logger.handlers[INDEX].setStream( open('/dev/ttys010', 'a') )

after registering a logging.FileHandler to your logger, you can reach into its internals and change the stream it outputs to "on the fly".

make sure INDEX accesses the right handler within the logger. if you only added a FileHandler, then it should be index 0.

Explained

Well first, following the logging documentation's suggested idiom, you would get a new logger instance named after the __name__ of whatever your specific package, module, class, or function:

#: class
>>> class A:
        def __init__(self):
            self.logger = logging.getLogger(self.__class__.__name__)
>>>A().logger
<Logger A (WARNING)>

#: function
>>> def func():
        logger = logging.getLogger(func.__name__)
        print(logger)
>>> func()
<Logger func (WARNING)>

#: module
>>> logger = logging.getLogger( __name__ )
>>> logger
<Logger __main__ (WARNING)>

#: package (e.g. a package named 'pkg', write this in '__init__.py')
>>> logger = logging.getLogger( __package__ )
>>> logger
<RootLogger pkg (WARNING)>

Next, if you've registered a logging.FileHandler handler for your logger, like so:

logger.addHandler( logging.FileHandler('/tmp/logfile.log', 'a') )

then you can you can change the file it outputs to by replacing the stream it outputs to:

INDEX = 0  # you will have to find the index position of the `FileHandler` you 
           # registered to this logger. I justed listed them with: `logger.handlers`
           # and picked the one I needed. if you only register one handler
           # then it should be at index 0, i.e the first one

#: change to new file location
logger.handlers[INDEX].setStream( open('/path/to/new/log/file.log', 'a') )

#: change to stdout or stderr
import sys
logger.handlers[INDEX].setStream( sys.stdout ) # or sys.stderr

#: change to another device
logger.handlers[INDEX].setStream( open('/dev/ttys010', 'a') )

If your curious, found this in a few minutes, by doing a bit of digging like so (in ipython and python interpreters):

>>> import logging
>>> logger = logging.getLogger( __name__ )
>>> logger.addHandler( logging.FileHandler('/tmp/logfile', 'a') )

>>> globals()
>>> dir(logger)

#: found that the logger has 'handlers' attribute
>>> dir(logger.handlers)
>>> logger.handlers

#: found that the FileHandler I registered earlier is at index: 0
>>> logger.handlers[0]
>>> dir(logger.handlers[0])

#: found that FileHandler has a dictionary '__dict__'
>>> logger.handlers[0].__dict__

#: found that FileHandler dict has 'baseFilename' attribute with the filename
#: i had set when registering the file handler
>>> logger.handlers[0].__dict__['baseFilename']

#: tried changing the file it points to
>>> logger.handlers[0].__dict__['baseFilename'] = '/tmp/logfile.log'
#: tried logging
>>> logger.info(f'hello world')
#: didn't work

#: found another interesting perhaps relevant attribute 'stream' in the
#: FileHandler dict
>>> logger.handlers[0].__dict__['stream']
>>> dir(logger.handlers[0].__dict__['stream'])
>>> logger.handlers[0].__dict__['stream'].__dict__
#: tried replacing the stream altogether
>>> logger.handlers[0].__dict__['stream'] = open('/tmp/logfile.log','a')
#: tried logging
>>> logger.info(f'hello world again')
#: it worked
>>> logger.info(f'hey it worked')

#: found another interesting perhaps relevant method 'setStream'
>>> logger.handlers[0].setStream( open('/tmp/otherlogfile.log','a') )
#: tried logging
>>> logger.info(f'hello world again')
#: it worked
>>> logger.info(f'hey it worked')

you can also change the name of the logger with:
```
logger.name = 'bla'
```
and more, see: dir(logger)

This looks like most direct approach, as long as you get the correct index. Less code == better answer. — pauljohn32, Sep 07 '22 at 05:17

How to change filehandle with Python logging on the fly with different classes and imports

5 Answers5

TL;DR

Explained

Linked