Loguru in Multiprocessing Workers

Loguru's sinks are thread-safe out of the box, which covers most single-process services. Once several processes write to the same sink — a pool of workers appending to one file, a prefork server's workers sharing a log path — the guarantees change, and the enqueue option and the process start method decide whether records arrive intact, arrive twice, or do not arrive at all. This page covers configuring sinks once, sharing them correctly under both fork and spawn, and draining them at shutdown. It is a task article under Loguru configuration and sinks, part of the modern Python logging libraries deep dive section, and it pairs with async logging with Loguru enqueue.

Prerequisites

pip install "loguru>=0.7.0,<0.8.0"

Implementation

Step 1 — Enable enqueue on sinks shared between processes. With enqueue=True, calls to the logger put records on a multiprocessing-safe queue, and a background thread in the process that added the sink takes them off and writes them. Every record reaches the sink through one writer, so lines never interleave and file rotation happens exactly once.

from loguru import logger

logger.remove()                                   # no default stderr sink
logger.add(
    "/var/log/app/worker.jsonl",
    serialize=True,                               # one JSON object per line
    enqueue=True,                                 # 1. one writer for every process
    rotation="64 MB",
    retention=5,
    level="INFO",
)

Step 2 — Configure in the parent, before starting workers. The sink with its queue and writer thread is created once, in the main process. Workers created afterwards by forking inherit the logger object and its reference to the queue, so their records flow to the parent's writer. Workers must not add their own sinks for the same destination, or the benefit is lost.

import multiprocessing as mp

def work(item: int) -> int:
    logger.info("processing item", item=item)     # goes through the parent's queue
    return item * 2

if __name__ == "__main__":
    ctx = mp.get_context("fork")
    with ctx.Pool(4) as pool:
        results = pool.map(work, range(100))
    logger.complete()                             # drain before the parent exits

Step 3 — Pass the logger explicitly under spawn. The spawn start method creates each child as a fresh interpreter that re-imports the main module and inherits nothing. Sinks configured in the parent do not exist there. Passing the configured logger to each worker through the pool's initialiser, and replacing the child's own logger with it, restores the path to the parent's queue.

import multiprocessing as mp
from loguru import logger

def init_worker(parent_logger):
    global logger
    logger = parent_logger                        # 2. the parent's queue-backed logger

def work(item: int) -> int:
    logger.info("processing item", item=item)
    return item * 2

if __name__ == "__main__":
    logger.remove()
    logger.add("/var/log/app/worker.jsonl", serialize=True, enqueue=True)
    ctx = mp.get_context("spawn")
    with ctx.Pool(4, initializer=init_worker, initargs=(logger,)) as pool:
        pool.map(work, range(100))
    logger.complete()

Expected Output: one file, whole lines, each carrying the worker's process identifier.

{"text": "processing item\n", "record": {"message": "processing item", "extra": {"item": 41}, "process": {"id": 18234, "name": "SpawnPoolWorker-2"}, "level": {"name": "INFO"}, "time": {"repr": "2026-09-18 14:02:11.408000+00:00"}}}

Step 4 — Drain the queue before exit. Records sit in the queue until the writer thread takes them, and the writer runs at its own pace rather than the workers'. If the parent exits before that happens, they are lost. logger.complete() blocks until everything enqueued so far has been written, and calling it at shutdown — after the pool has finished — makes sure the final records reach the file.

Step 5 — Under a prefork server, prefer standard output. A web server's workers are separate processes too, but the simplest correct arrangement there is not a shared file at all: each worker writes to standard output, the platform captures each process's stream, and nothing is shared. Loguru's standard error sink is replaced with a standard output sink in each worker, with enqueue optional — it adds a background writer, which keeps a slow pipe off request threads, as described in async logging with Loguru enqueue.

Why the standard library faces the same problem

None of this is peculiar to Loguru. The standard library's logging has exactly the same issue with several processes appending to one file, and the same solution: a queue feeding a single writer, with QueueHandler in the workers and a QueueListener in the parent, as described in thread-safe logging in multiprocessing.

What Loguru adds is packaging. The single enqueue=True flag sets up the queue, the writer thread and the handoff, where the standard library needs three objects wired together by hand. The price is that the mechanism is less visible: developers who have not read about it may not realise that the writer thread lives in the process that called add, which is why adding the sink in each worker — rather than once in the parent — quietly defeats it.

The fork-versus-spawn distinction applies to both libraries equally. Anything relying on inheriting configured state from the parent works under fork and fails under spawn. Code intended to run on more than one platform, or to survive a future change in the default start method, should pass whatever the workers need explicitly rather than relying on inheritance.

A further option sidesteps the question for pools that do not need a shared file: each worker logs to standard output with its process identifier in every record, and whatever collects standard output keeps the streams separate. This is how prefork web servers are usually run, and it works for task pools just as well when the output is being collected by a platform rather than read from a local file.

Context that identifies the worker

Once records from many processes land in one file, telling them apart becomes the next problem, and Loguru's record already carries most of what is needed.

Every record includes the process identifier and name, which is enough to separate workers within one run. Across runs it is not: process identifiers are reused, so a pid alone cannot distinguish today's worker 18234 from yesterday's. Binding a run identifier once in the parent — logger = logger.bind(run_id=...) before the pool starts — gives every record from every worker in that run a common key, and the process identifier then distinguishes workers within it.

For task pools, binding the task's own identifier inside the worker function, with logger.contextualize(task_id=...) around the work, attaches it to every record the task produces, including records from library code called within it. Because contextualize uses context variables, it is scoped correctly even when a worker runs several tasks in sequence, and it never leaks from one task to the next.

These values travel through the enqueue queue with the record, which means they must be picklable. Plain strings and integers always are; bound objects — a request, a database session, a client — may not be, and fail at the moment of logging rather than at configuration time. Binding identifiers rather than objects avoids the problem entirely and is better practice regardless, for the reasons set out in logging personal data safely.

Configuration options

Verification

Run a pool that logs many long records and check the file for interleaved or duplicated lines and a single rotation sequence.

import json
bad, total = 0, 0
for line in open("/var/log/app/worker.jsonl"):
    total += 1
    try:
        json.loads(line)
    except json.JSONDecodeError:
        bad += 1
print(f"{total} lines, {bad} malformed")

Expected Output: every line whole.

400000 lines, 0 malformed

Malformed lines mean more than one process is writing to the file directly — a worker added its own sink, or the sink was added without enqueue.

Common mistakes

A file sink added in every worker. Error signature: interleaved lines and several rotated files. Root cause: each process writing and rotating independently. Remediation: add once in the parent with enqueue=True.

Relying on inheritance under spawn. Error signature: workers that log nothing to the file. Root cause: spawned children start with Loguru's defaults. Remediation: pass the logger through the pool initialiser.

No complete() at shutdown. Error signature: the last records before exit missing. Root cause: the queue abandoned with records in it. Remediation: call logger.complete() after the pool finishes.

A shared file under a web server. Error signature: file contention and rotation problems in production only. Root cause: server workers sharing a path. Remediation: standard output per worker.

enqueue with unpicklable extras. Error signature: an error when logging objects bound with bind. Root cause: records crossing a process boundary must be picklable. Remediation: bind plain values — strings, numbers — rather than objects.

Frequently Asked Questions