58

I'm trying to replace an ad-hoc logging system with Python's logging module. I'm using the logging system to output progress information for a long task on a single line so you can tail the log or watch it in a console. I've done this by having a flag on my logging function which suppresses the newline for that log message and build the line piece by piece.

All the logging is done from a single thread so there's no serialisation issues.

Is it possible to do this with Python's logging module? Is it a good idea?

vvvvv
  • 25,404
  • 19
  • 49
  • 81
Peter Graham
  • 11,323
  • 7
  • 40
  • 42
  • If you want to log multiple things in the same line consider not using `logging` for it but use a good old file object and write to it directly (and to a different file than the logfile used for other, line-based, things). – ThiefMaster Aug 24 '11 at 00:02
  • 1
    @ThiefMaster I'm writing to a file and stdout directly right now. It feels like I'm rewriting logging though, so I'd rather use logging if it ends up being less effort. – Peter Graham Aug 24 '11 at 00:52

5 Answers5

71

If you wanted to do this you can change the logging handler terminator. I'm using Python 3.4. This was introduced in Python 3.2 as stated by Ninjakannon.

handler = logging.StreamHandler()
handler.terminator = ""

When the StreamHandler writes it writes the terminator last.

justengel
  • 6,132
  • 4
  • 26
  • 42
  • 13
    This was [introduced in Python 3.2](https://docs.python.org/3/library/logging.handlers.html#streamhandler). – Ninjakannon Apr 24 '16 at 12:15
  • 4
    How do I integrate this with the logger? – Abdel Aleem Aug 21 '20 at 08:06
  • 1
    You don't. The handlers are the object that writes the information to a file, socket, stream, or other. Most handlers inherit from the StreamHandler. The terminator variable is a class level variable, so you can change this for all handlers with `logging.StreamHandler.terminator = ""`. This should change all handlers except ones where the terminator was explicitly set. You could also loop through the logger handlers and set the terminator `for handler in logger.handlers:`... – justengel Aug 21 '20 at 12:36
  • 1
    See [logging-cookbook](https://docs.python.org/3/howto/logging-cookbook.html) in the docs for examples of using handlers. Looks like this answer needs something like `logging.getLogger('').addHandler(handler)` to be complete. – idbrii Apr 21 '22 at 22:55
16

Let's start with your last question: No, I do not believe it's a good idea. IMO, it hurts the readability of the logfile in the long run.

I suggest sticking with the logging module and using the '-f' option on your 'tail' command to watch the output from the console. You will probably end up using the FileHandler. Notice that the default argument for 'delay' is False meaning the output won't be buffered.

If you really needed to suppress newlines, I would recommend creating your own Handler.

jonEbird
  • 408
  • 3
  • 8
13

The new line, \n, is inserted inside the StreamHandler class.

If you're really set on fixing this behaviour, then here's an example of how I solved this by monkey patching the emit(self, record) method inside the logging.StreamHandler class.

A monkey patch is a way to extend or modify the run-time code of dynamic languages without altering the original source code. This process has also been termed duck punching.

Here is the custom implementation of emit() that omits line breaks:

def customEmit(self, record):
    # Monkey patch Emit function to avoid new lines between records
    try:
        msg = self.format(record)
        if not hasattr(types, "UnicodeType"): #if no unicode support...
            self.stream.write(msg)
        else:
            try:
                if getattr(self.stream, 'encoding', None) is not None:
                    self.stream.write(msg.encode(self.stream.encoding))
                else:
                    self.stream.write(msg)
            except UnicodeError:
                self.stream.write(msg.encode("UTF-8"))
        self.flush()
    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        self.handleError(record)

Then you would make a custom logging class (in this case, subclassing from TimedRotatingFileHandler).

class SniffLogHandler(TimedRotatingFileHandler):
    def __init__(self, filename, when, interval, backupCount=0,
                 encoding=None, delay=0, utc=0):

        # Monkey patch 'emit' method
        setattr(StreamHandler, StreamHandler.emit.__name__, customEmit)

        TimedRotatingFileHandler.__init__(self, filename, when, interval,
                                          backupCount, encoding, delay, utc)

Some people might argue that this type of solution is not Pythonic, or whatever. It might be so, so be careful.

Also, be aware that this will globally patch SteamHandler.emit(...), so if you are using multiple logging classes, then this patch will affect the other logging classes as well!

Check out these for further reading:

Hope that helps.

Community
  • 1
  • 1
pjama
  • 3,014
  • 3
  • 26
  • 27
10

Python 3.5.9

class MFileHandler(logging.FileHandler):
  """Handler that controls the writing of the newline character"""

  special_code = '[!n]'

  def emit(self, record) -> None:

    if self.special_code in record.msg:
      record.msg = record.msg.replace( self.special_code, '' )
      self.terminator = ''
    else:
      self.terminator = '\n'

    return super().emit(record)

Then

fHandler = MFileHandler(...)

Example:

# without \n
log.info( 'waiting...[!n]' )
 ...
log.info( 'OK' )

# with \n
log.info( 'waiting...' )
 ...
log.info( 'OK' )

log.txt:

waiting...OK
waiting...
OK
kernelplv
  • 111
  • 2
  • 4
2

I encountered a need to log a certain section in a single line as I iterated through a tuple, but wanted to retain the overall logger.

Collected the output first into a single string and later sent it out to logger once I was out of the section. Example of concept 

for fld in object._fields: 
  strX = (' {} --> {} ').format(fld, formattingFunction(getattr(obj,fld)))
debugLine += strX
logger.debug('{}'.format(debugLine))
Rembau Times
  • 21
  • 5