Python multiprocessing log file is empty

Question

Below is my dummy script. It works fine, but the log file is empty even post running. What is wrong? I am assuming it to be related to the multiprocessing logger but not able to figure out yet.

import os
import multiprocessing
import traceback
import logging

def get_ids():
    return [i for i in range(100)]

def function_name(ids):
    if ids[0] % 4 == 0:
        raise Exception(f"Error processing chunk starting with id {ids[0]}")
    
    # Do some processing here
    print(f"Processing chunk starting with id {ids[0]}")
    return

def process_chunk(chunk):
    logger = multiprocessing.get_logger()
    try:
        function_name(chunk)
    except Exception as e:
        logger.exception(f"Error processing chunk {chunk}")

if __name__ == "__main__":
    log_path = os.path.abspath(os.path.join(os.path.dirname(__file__), 'example.log'))
    logging.basicConfig(filename=log_path, level=logging.DEBUG)

    ids = get_ids()
    chunks = [ids[i:i + 5] for i in range(0, len(ids), 5)]

    pool = multiprocessing.Pool(processes=8)
    for chunk in chunks:
        pool.apply(process_chunk, args=(chunk,))
    pool.close()
    pool.join()

I am raising exception manually if you see, atleast that should be captured? I am using debug level in logging!! — Anant, May 01 '23 at 13:51

score 0 · Answer 1 · answered May 01 '23 at 14:26

Try the following:

def process_chunk(chunk):
    logger = multiprocessing.get_logger()
    logger.setLevel(logging.DEBUG)

    handler = logging.FileHandler(
        os.path.join(os.path.dirname(__file__), "76147452.log")
    )

    # Avoid duplicated messages in the output
    if len(logger.handlers) == 0:
        logger.addHandler(handler)

    try:
        function_name(chunk)
    except Exception as e:
       logger.exception(f"Error processing chunk {chunk}")

Also, see Logging to a single file from multiple processes.

score 0 · Answer 2 · answered May 01 '23 at 14:27

I'm new to this topic, but it seems like this problem is mentioned in the docs for multiprocessing.get_logger(): "Messages sent to this logger will not by default propagate to the root logger." So you just need to enable propagation:

def process_chunk(chunk):
    try:
    ...

if __name__ == "__main__":
    log_path = ...
    logging.basicConfig(...)

    logger = multiprocessing.get_logger()
    logger.propagate = True

    ...

I'm also getting logger in the global scope here, which starts its logging earlier, but I'm not sure if it has any other effect.

Python multiprocessing log file is empty

2 Answers2