Thread-Safe and Multiprocessing-Safe Logging in Python

Writing logs from a process pool requires funnelling every record through a single writer, because file handlers held independently by multiple processes corrupt output when their writes interleave. This guide is part of the Python Logging Fundamentals and Structured Data guide and extends the context variables and thread safety guide with the cross-process case, where the per-thread locking that protects intra-process logging no longer applies.

Prerequisites

Everything here is standard library: logging.handlers.QueueHandler, logging.handlers.QueueListener, and multiprocessing. No install is required. The intra-process, non-blocking version of this pattern is covered in non-blocking logging with QueueHandler; this page extends it across the process boundary.

# These imports are all you need.
import logging
import logging.handlers
import multiprocessing

Implementation

The central insight is that thread safety and process safety are different problems with different solutions, even though both surface as garbled log output. A threading.Lock makes a single handler safe across threads, but each process gets its own copy of that lock and its own file descriptor, so file handlers are not multiprocess-safe. Standard library handlers acquire a per-handler lock around their emit call, which serializes threads within one interpreter; that lock means nothing to a separate process holding an independent handler object. Two processes can therefore both be inside emit at the same instant, and if their writes exceed the filesystem's atomic-write size or are flushed separately, the bytes interleave into corrupt lines. Rotating file handlers are worse, because a rename during rollover in one process pulls the file out from under the others. The fix is to have workers only enqueue, and let exactly one process write.

Step 1 — Create a shared queue in the parent. Use multiprocessing.Queue, which is picklable and proxied across processes. A plain queue.Queue would not survive being passed to a worker.

import multiprocessing

log_queue: "multiprocessing.Queue" = multiprocessing.Queue(-1)

Step 2 — Configure each worker to log into the queue. Pass the queue to the worker initializer. Inside the worker, attach a single QueueHandler to the root logger and remove any inherited handlers so nothing writes to a file directly.

import logging
import logging.handlers


def worker_init(queue: "multiprocessing.Queue") -> None:
    root = logging.getLogger()
    root.handlers.clear()              # drop any inherited file handlers
    root.addHandler(logging.handlers.QueueHandler(queue))
    root.setLevel(logging.INFO)


def do_work(item: str) -> None:
    # Pass request-scoped context explicitly; contextvars do not cross processes.
    logging.getLogger("worker").info("processing %s", item)

Step 3 — Run the listener in the parent only. The QueueListener owns the real handlers, here a stream handler with a JSON-friendly formatter. It runs on a background thread inside the parent process, so there is a single writer.

import logging
import logging.handlers


def start_listener(queue: "multiprocessing.Queue") -> logging.handlers.QueueListener:
    handler = logging.StreamHandler()
    handler.setFormatter(
        logging.Formatter('{"level":"%(levelname)s","proc":"%(processName)s","msg":"%(message)s"}')
    )
    listener = logging.handlers.QueueListener(queue, handler, respect_handler_level=True)
    listener.start()
    return listener

Step 4 — Wire the pool to the queue and drain on shutdown. Use the pool initializer so every worker installs the QueueHandler exactly once at startup. Stop the listener last so buffered records flush.

import multiprocessing

if __name__ == "__main__":
    log_queue = multiprocessing.Queue(-1)
    listener = start_listener(log_queue)
    try:
        with multiprocessing.Pool(
            processes=3,
            initializer=worker_init,
            initargs=(log_queue,),
        ) as pool:
            pool.map(do_work, ["a", "b", "c"])
    finally:
        listener.stop()   # flush remaining records, then join the thread

Expected Output:

{"level":"INFO","proc":"SpawnPoolWorker-1","msg":"processing a"}
{"level":"INFO","proc":"SpawnPoolWorker-2","msg":"processing b"}
{"level":"INFO","proc":"SpawnPoolWorker-3","msg":"processing c"}

Each line is whole because only the parent's listener touches the destination. Ordering across processes is not guaranteed, but no record is ever split. Note that the heavy work, formatting the record into a string and writing it to disk, now happens once in the listener thread rather than in every worker. The workers do only the cheap part: constructing the LogRecord and dropping it on the queue. This both removes the I/O contention and keeps worker CPU focused on real work, which is the same decoupling principle that motivates the intra-process queue pattern, applied across the process boundary.

One subtlety with multiprocessing.Queue is that it is backed by a feeder thread and an OS pipe. A worker that exits abruptly while records are still buffered in its feeder thread can lose those records, so let workers finish cleanly and join the pool before stopping the listener. The Pool context manager handles the join for you when the with block exits normally.

Context across the process boundary

Context variables do not cross processes. With the spawn start method, a worker starts a fresh interpreter that inherits none of the parent's contextvars. Even with fork, mutations made after the fork are invisible to the parent. The using contextvars for request tracing guide covers the intra-process model; across processes you must pass correlation data explicitly, for example as an argument to do_work and then attach it with the extra parameter so it lands on the record before it enters the queue.

def do_work(item: str, request_id: str) -> None:
    logging.getLogger("worker").info("processing %s", item, extra={"request_id": request_id})

Configuration options

Verification

Assert that workers carry exactly one handler and that it is a QueueHandler. This catches the most common misconfiguration, an inherited file handler still attached in the child.

def assert_worker_logging() -> None:
    handlers = logging.getLogger().handlers
    assert len(handlers) == 1, handlers
    assert isinstance(handlers[0], logging.handlers.QueueHandler)

Expected Output: the function returns without raising. If it raises AssertionError, a handler other than QueueHandler is still attached in the worker and will write to disk concurrently.

To confirm no interleaving under load, run the pool with thousands of short messages and check the output file has the expected line count with no truncated JSON, for example with wc -l and a JSON-per-line parse pass. A useful stress test is to make each worker emit a record whose message contains its own process identifier and a monotonic counter, then after the run assert that every counter value for every worker appears exactly once and that each parsed line is valid JSON. If any line fails to parse or a counter is missing, a writer other than the listener is still touching the file, which points straight back to an inherited handler in a worker.

Common mistakes

Passing a queue.Queue to workers. A standard-library queue lives in one process's memory and cannot be shared. Workers receive a useless copy and their records vanish. Use multiprocessing.Queue or a manager queue.

Leaving file handlers attached in workers. If a child inherits or re-adds a FileHandler, it writes directly to disk alongside the listener, reintroducing the interleaving you set out to fix. Clear handlers in the worker initializer and attach only a QueueHandler.

Forgetting to stop the listener. Records sit in the queue or the listener's buffer until listener.stop() flushes them. Skipping it on shutdown silently drops the tail of your logs. Always stop the listener in a finally block.

Frequently Asked Questions