Using LogRecordFactory in python to add custom fields for logging

Question

I am trying to add a custom field in my logging using LogRecordFactory. I am repeatedly calling a class and every time I do that, I want to set the custom_attribute in the init module so the remainder of the code within the class will have this attribute. But I cannot get this to work. I found the following which works, but its static.

import logging

old_factory = logging.getLogRecordFactory()

def record_factory(*args, **kwargs):
    record = old_factory(*args, **kwargs)
    record.custom_attribute = "whatever"
    return record



logging.basicConfig(format="%(custom_attribute)s - %(message)s")
logging.setLogRecordFactory(record_factory)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")

This will output correctly:

whatever - test

However, my use case is that the custom_attribute will vary. Every time I call a specific function, I want to change this. So it seems like record_factory needs another parameter passed to it so it can then return the correct record with the new parameter. But I cant figure it out. I have tried adding a parameter to the function, but when I make the call I get:

TypeError: __init__() missing 7 required positional arguments: 'name', 'level', 'pathname', 'lineno', 'msg', 'args', and 'exc_info'

I think this has something to do with the *args and **kwargs but I don't really know. Also, why are there no parenthesis after record_factory when its called by logging.setLogRecordFactory? I have never seen a function work like this.

score 4 · Answer 1 · answered Mar 23 '21 at 13:46

You can try to use closure:

import logging

old_factory = logging.getLogRecordFactory()

def record_factory_factory(context_id):
    def record_factory(*args, **kwargs):
        record = old_factory(*args, **kwargs)
        record.custom_attribute = context_id
        return record
    return record_factory


logging.basicConfig(format="%(custom_attribute)s - %(message)s")
logging.setLogRecordFactory(record_factory_factory("whatever"))
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")

logging.setLogRecordFactory(record_factory_factory("whatever2"))
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logging.debug("test")

result:

$ python3 log_test.py                                                          
whatever - test
whatever2 - test

FlorianK · Answer 2 · 2020-07-30T15:02:28.673

I stumbled upon this question while I was trying to do something similar. This is how I solved it, assuming that you want to add something called xyz to every log line (further explanation below):

import logging
import threading
 

thread_local = threading.local()
 

def add_xyz_to_logrecords(xyz):
    factory = logging.getLogRecordFactory()
    if isinstance(factory, XYZLogFactory):
        factory.set_xyz(xyz)
    else:
        logging.setLogRecordFactory(XYZLogFactory(factory, xyz))
 

class XYZLogFactory():
    def __init__(self, original_factory, xyz):
        self.original_factory = original_factory
        thread_local.xyz = xyz

    def __call__(self, *args, **kwargs):
        record = self.original_factory(*args, **kwargs)
        try:
            record.xyz = thread_local.xyz
        except AttributeError:
            pass
        return record

   def set_xyz(self, xyz):
        thread_local.xyz = xyz

Here I've created a callable class XYZLogFactory, that remembers what the current value of xyz is, and also remembers what the original LogRecordFactory was. When called as a function, it creates a record using the original LogRecordFactory, and adds an xyz attribute with the current value. The thread_local is to make it thread-safe, but for an easier version, you could just use an attribute on the XYZLogFactory:

class XYZLogFactory():
    def __init__(self, original_factory, xyz):
        self.original_factory = original_factory
        self.xyz = xyz

    def __call__(self, *args, **kwargs):
        record = self.original_factory(*args, **kwargs)
        record.xyz = self.xyz
        return record

   def set_xyz(self, xyz):
        self.xyz = xyz

In my very first attempt (not shown here), I did not store the original factory, but stored it implicitly in the new LogRecordFactury using a closure. However, after a while that led to a RecursionError, because it kept calling the previous factory, which called the previous factory, etc.

Regarding your last question: there are no parentheses because the function is not called here. Instead it's passed to the logging.setLogRecordFactory, which saves it in a variable somewhere, and then calls that someplace else. If you want more information you can google something like 'functions as first class citizens'. Easy example:

x = str  # Assign to x the function that gives string representation of object
x(1)  # outputs the string representation of 1, same as if you'd called str(1)
> '1'

Using LogRecordFactory in python to add custom fields for logging

2 Answers2