Making Python loggers output all messages to stdout in addition to log file

Question

Is there a way to make Python logging using the logging module automatically output things to stdout in addition to the log file where they are supposed to go? For example, I'd like all calls to logger.warning, logger.critical, logger.error to go to their intended places but in addition always be copied to stdout. This is to avoid duplicating messages like:

mylogger.critical("something failed")
print("something failed")

Please check this answer https://stackoverflow.com/questions/9321741/printing-to-screen-and-writing-to-a-file-at-the-same-time — SeF, Feb 15 '18 at 17:08

Martijn Pieters · Accepted Answer · 2018-11-08T11:47:10.747

901

All logging output is handled by the handlers; just add a logging.StreamHandler() to the root logger.

Here's an example configuring a stream handler (using stdout instead of the default stderr) and adding it to the root logger:

import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)

edited Nov 08 '18 at 11:47

answered Dec 27 '12 at 17:12

Martijn Pieters

1,048,767
296
4,058
3,343

12

That's fine but if it's already redirected to a file how can I have it be printed to ``stdout`` in addition? – Dec 27 '12 at 17:27
78

@user248237: By adding a *new* handler as illustrated. New handlers do not replace the existing handlers, they *also* get to process the log entries. – Martijn Pieters Dec 27 '12 at 17:28
@MartijnPieters is there a way to add a string to every log statement printed out? – praxmon Dec 03 '15 at 06:08
7

@PrakharMohanSrivastava I'd guess you can just add it to the string passed into `logging.Formatter`. – A.Wan Jan 12 '16 at 21:39
@MartijnPieters Why setLevel is done multiple times both in StreamHandler and at root level? – himanshu219 Oct 25 '18 at 08:55
3

@himanshu219: the logger has a level, and the handler has a level. The logger will handle messages of that level and higher, and the handler will handle messages of that level and higher. It lets you differentiate between different loggers and different handlers. – Martijn Pieters Oct 25 '18 at 10:31
1

@MartijnPieters my point was what is the use case here? If we have a single handler wouldn't it be sufficient to set level only in root logger. Also another point is even if one has multiple handlers wouldn't it makes sense to set level only at handler level not at logger level.The only use case I think is that setting level at logger level sets the default level for those handlers which do not set any level. – himanshu219 Oct 26 '18 at 10:39
8

@himanshu219: the use case is that as soon as you start adding multiple handlers, you usually want to differentiate. DEBUG to the console, WARNING and up to a file, etc. – Martijn Pieters Oct 26 '18 at 11:01
Do you need to do `root = logging.getLogger()` if you use `logging.basicConfig()`? – Lou Jan 28 '21 at 14:21
1

@Lou: if you can get the desired logging configuration set up with `logging.basicConfig()`, then no, you don't need to fetch a reference to the root logger. – Martijn Pieters Jan 30 '21 at 13:09
how about catching subprocess log calls back to the parent process as variables? – Vaidøtas I. Feb 26 '21 at 23:00
@VaidøtasI. that's well outside the scope of this question/answer pair, and depends on a lot of factors. – Martijn Pieters Feb 27 '21 at 17:07
Hi, great answer, but seeing it's been 5 years since updated, is it better now to do it [differently](https://stackoverflow.com/a/46098711/19674402)? – PythonForEver Apr 07 '23 at 09:37
@PythonForEver: if you already have a configured root logger, then `basicConfig()` will not touch the handlers on it (unless you set `force=True`, and then it'll replace _all_ handlers). – Martijn Pieters Jun 10 '23 at 22:18

score 788 · Answer 2 · edited Feb 15 '23 at 10:44

788

The simplest way to log to stdout using basicConfig:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)

edited Feb 15 '23 at 10:44

Flimm

136,138
45
251
267

answered Jan 28 '15 at 14:39

Eyal

8,219
1
13
7

91

Hm, but this isn't logged to a file, right? The question was how to do logging to file _and_ to console. – Weidenrinde May 07 '15 at 09:17
Ref link: [Python3 Docs: Logging.basicConfig](https://docs.python.org/3/library/logging.html#logging.basicConfig) – Czechnology Nov 06 '17 at 21:06
7

In Python 3 at least, it looks like omitting `stream=sys.stdout` still works for logging to the console for me. – Taylor D. Edmiston May 20 '18 at 02:29
6

@TaylorEdmiston Yeah, but it's the stderr stream AFAIK. Try redirecting the output from the shell. – Sorin May 30 '18 at 14:24
3

OK. This doesn't answer both: logging to file and to console, but it was nice to find what I needed in 3 lines or less. – Steve3p0 Jul 23 '19 at 19:52
it's a bit late & i'm not professional at logging, but i think we could get anything into a file using `with open('logfile.txt', 'a+', encoding='utf-8') as f:` then do `print(logging ..., file=f)` – Ali Abdi Feb 01 '22 at 18:32
@Sorin can confirm it would stream to stdout. – sfsf9797 Oct 24 '22 at 12:43

score 146 · Answer 3 · edited Jul 20 '22 at 10:23

You could create two handlers for file and stdout and then create one logger with handlers argument to basicConfig. It could be useful if you have the same log_level and format output for both handlers:

import logging
import sys

file_handler = logging.FileHandler(filename='tmp.log')
stdout_handler = logging.StreamHandler(stream=sys.stdout)
handlers = [file_handler, stdout_handler]

logging.basicConfig(
    level=logging.DEBUG, 
    format='[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s',
    handlers=handlers
)

logger = logging.getLogger('LOGGER_NAME')

score 83 · Answer 4 · edited Apr 05 '17 at 13:03

It's possible using multiple handlers.

import logging
import auxiliary_module

# create logger with 'spam_application'
log = logging.getLogger('spam_application')
log.setLevel(logging.DEBUG)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
log.addHandler(fh)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
ch.setFormatter(formatter)
log.addHandler(ch)

log.info('creating an instance of auxiliary_module.Auxiliary')
a = auxiliary_module.Auxiliary()
log.info('created an instance of auxiliary_module.Auxiliary')

log.info('calling auxiliary_module.Auxiliary.do_something')
a.do_something()
log.info('finished auxiliary_module.Auxiliary.do_something')

log.info('calling auxiliary_module.some_function()')
auxiliary_module.some_function()
log.info('done with auxiliary_module.some_function()')

# remember to close the handlers
for handler in log.handlers:
    handler.close()
    log.removeFilter(handler)

Please see: https://docs.python.org/2/howto/logging-cookbook.html

Wonderful answer, albeit a bit messy. Love how you show how to use different levels and formats for streams and files. +1, but +2 in spirit. — The Unfun Cat, Jun 23 '15 at 21:30
For me this did not work without the `sys.stdout` parameter in `ch = logging.StreamHandler()` — veuncent, Oct 14 '19 at 13:02

score 43 · Answer 5 · edited Mar 29 '22 at 11:23

Here is a solution based on the powerful but poorly documented logging.config.dictConfig method. Instead of sending every log message to stdout, it sends messages with log level ERROR and higher to stderr and everything else to stdout. This can be useful if other parts of the system are listening to stderr or stdout.

import logging
import logging.config
import sys

class _ExcludeErrorsFilter(logging.Filter):
    def filter(self, record):
        """Only lets through log messages with log level below ERROR ."""
        return record.levelno < logging.ERROR


config = {
    'version': 1,
    'filters': {
        'exclude_errors': {
            '()': _ExcludeErrorsFilter
        }
    },
    'formatters': {
        # Modify log message format here or replace with your custom formatter class
        'my_formatter': {
            'format': '(%(process)d) %(asctime)s %(name)s (line %(lineno)s) | %(levelname)s %(message)s'
        }
    },
    'handlers': {
        'console_stderr': {
            # Sends log messages with log level ERROR or higher to stderr
            'class': 'logging.StreamHandler',
            'level': 'ERROR',
            'formatter': 'my_formatter',
            'stream': sys.stderr
        },
        'console_stdout': {
            # Sends log messages with log level lower than ERROR to stdout
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'my_formatter',
            'filters': ['exclude_errors'],
            'stream': sys.stdout
        },
        'file': {
            # Sends all log messages to a file
            'class': 'logging.FileHandler',
            'level': 'DEBUG',
            'formatter': 'my_formatter',
            'filename': 'my.log',
            'encoding': 'utf8'
        }
    },
    'root': {
        # In general, this should be kept at 'NOTSET'.
        # Otherwise it would interfere with the log levels set for each handler.
        'level': 'NOTSET',
        'handlers': ['console_stderr', 'console_stdout', 'file']
    },
}

logging.config.dictConfig(config)

had to rename the logger to an empty string to actually get the root logger. Otherwise very helpful, thanks ! — Newtopian, Sep 03 '19 at 15:26
whoa, never realized the existence of `dictConfig` before!! much gratitude!!! — pepoluan, Sep 30 '20 at 09:53
up for dictConfig usage which allows loading logging from a config file more easily — rkachach, Mar 29 '22 at 11:23
@rkachach the idea of storing logger config in a yaml file seems more robust as it allows code reusage — ivan866, Oct 31 '22 at 20:22
This is how django logging is setup, it allows for some finegrained and powerful logging configurations... — miller the gorilla, Mar 21 '23 at 10:09

score 39 · Answer 6 · edited Jun 24 '19 at 16:54

39

The simplest way to log to file and to stderr:

import logging

logging.basicConfig(filename="logfile.txt")
stderrLogger=logging.StreamHandler()
stderrLogger.setFormatter(logging.Formatter(logging.BASIC_FORMAT))
logging.getLogger().addHandler(stderrLogger)

edited Jun 24 '19 at 16:54

Stevoisiak

23,794
27
122
225

answered May 07 '15 at 09:22

Weidenrinde

2,152
1
20
21

This doesn't show labels INFO, DEBUG, and ERROR before the logging message in the console. It does show those labels in the file. Any ideas to also show the labels in the console ? – JahMyst Jan 20 '16 at 20:22
@JahMyst you have to change the `format` parameter of `root` logger's formatter – ivan866 Oct 31 '22 at 20:34

Milovan Tomašević · Answer 7 · 2021-01-31T22:56:04.303

For more detailed explanations - great documentation at that link. For example: It's easy, you only need to set up two loggers.

import sys
import logging

logger = logging.getLogger('')
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler('my_log_info.log')
sh = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter('[%(asctime)s] %(levelname)s [%(filename)s.%(funcName)s:%(lineno)d] %(message)s', datefmt='%a, %d %b %Y %H:%M:%S')
fh.setFormatter(formatter)
sh.setFormatter(formatter)
logger.addHandler(fh)
logger.addHandler(sh)

def hello_logger():
    logger.info("Hello info")
    logger.critical("Hello critical")
    logger.warning("Hello warning")
    logger.debug("Hello debug")

if __name__ == "__main__":
    print(hello_logger())

Output - terminal:

[Mon, 10 Aug 2020 12:44:25] INFO [TestLoger.py.hello_logger:15] Hello info
[Mon, 10 Aug 2020 12:44:25] CRITICAL [TestLoger.py.hello_logger:16] Hello critical
[Mon, 10 Aug 2020 12:44:25] WARNING [TestLoger.py.hello_logger:17] Hello warning
[Mon, 10 Aug 2020 12:44:25] DEBUG [TestLoger.py.hello_logger:18] Hello debug
None

Output - in file:

UPDATE: color terminal

Package:

pip install colorlog

Code:

import sys
import logging
import colorlog

logger = logging.getLogger('')
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler('my_log_info.log')
sh = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter('[%(asctime)s] %(levelname)s [%(filename)s.%(funcName)s:%(lineno)d] %(message)s', datefmt='%a, %d %b %Y %H:%M:%S')
fh.setFormatter(formatter)
sh.setFormatter(colorlog.ColoredFormatter('%(log_color)s [%(asctime)s] %(levelname)s [%(filename)s.%(funcName)s:%(lineno)d] %(message)s', datefmt='%a, %d %b %Y %H:%M:%S'))
logger.addHandler(fh)
logger.addHandler(sh)

def hello_logger():
    logger.info("Hello info")
    logger.critical("Hello critical")
    logger.warning("Hello warning")
    logger.debug("Hello debug")
    logger.error("Error message")

if __name__ == "__main__":
    hello_logger()

output:

Recommendation:

Complete logger configuration from INI file, which also includes setup for stdout and debug.log:

handler_file
- level=WARNING
handler_screen
- level=DEBUG

score 15 · Answer 8 · answered Jan 23 '19 at 12:32

15

Since no one has shared a neat two liner, I will share my own:

logging.basicConfig(filename='logs.log', level=logging.DEBUG, format="%(asctime)s:%(levelname)s: %(message)s")
logging.getLogger().addHandler(logging.StreamHandler())

answered Jan 23 '19 at 12:32

Lexander

184
2
10

score 2 · Answer 9 · answered Apr 02 '19 at 20:00

Here's an extremely simple example:

import logging
l = logging.getLogger("test")

# Add a file logger
f = logging.FileHandler("test.log")
l.addHandler(f)

# Add a stream logger
s = logging.StreamHandler()
l.addHandler(s)

# Send a test message to both -- critical will always log
l.critical("test msg")

The output will show "test msg" on stdout and also in the file.

score 1 · Answer 10 · answered Jan 07 '23 at 12:32

1

You should use tee. Configure the app to write to stdout, and run

python3 app.py | tee log.txt

Then you'll get the log messages in your stdout as well as in log.txt.

answered Jan 07 '23 at 12:32

ninpnin

155
4

score 0 · Answer 11 · answered Jul 28 '20 at 13:18

I simplified my source code (whose original version is OOP and uses a configuration file), to give you an alternative solution to @EliasStrehle's one, without using the dictConfig (thus easiest to integrate with existing source code):

import logging
import sys


def create_stream_handler(stream, formatter, level, message_filter=None):
    handler = logging.StreamHandler(stream=stream)
    handler.setLevel(level)
    handler.setFormatter(formatter)
    if message_filter:
        handler.addFilter(message_filter)
    return handler


def configure_logger(logger: logging.Logger, enable_console: bool = True, enable_file: bool = True):
    if not logger.handlers:
        if enable_console:
            message_format: str = '{asctime:20} {name:16} {levelname:8} {message}'
            date_format: str = '%Y/%m/%d %H:%M:%S'
            level: int = logging.DEBUG
            formatter = logging.Formatter(message_format, date_format, '{')

            # Configures error output (from Warning levels).
            error_output_handler = create_stream_handler(sys.stderr, formatter,
                                                         max(level, logging.WARNING))
            logger.addHandler(error_output_handler)

            # Configures standard output (from configured Level, if lower than Warning,
            #  and excluding everything from Warning and higher).
            if level < logging.WARNING:
                standard_output_filter = lambda record: record.levelno < logging.WARNING
                standard_output_handler = create_stream_handler(sys.stdout, formatter, level,
                                                                standard_output_filter)
                logger.addHandler(standard_output_handler)

        if enable_file:
            message_format: str = '{asctime:20} {name:16} {levelname:8} {message}'
            date_format: str = '%Y/%m/%d %H:%M:%S'
            level: int = logging.DEBUG
            output_file: str = '/tmp/so_test.log'

            handler = logging.FileHandler(output_file)
            formatter = logging.Formatter(message_format, date_format, '{')
            handler.setLevel(level)
            handler.setFormatter(formatter)
            logger.addHandler(handler)

This is a very simple way to test it:

logger: logging.Logger = logging.getLogger('MyLogger')
logger.setLevel(logging.DEBUG)
configure_logger(logger, True, True)
logger.debug('Debug message ...')
logger.info('Info message ...')
logger.warning('Warning ...')
logger.error('Error ...')
logger.fatal('Fatal message ...')

Making Python loggers output all messages to stdout in addition to log file

11 Answers11

UPDATE: color terminal

Recommendation:

Linked