Async and Non-Blocking Logging with Loguru enqueue

A logging call that writes to a slow file, a network socket, or a sink under lock contention stalls the thread that made it; Loguru's enqueue=True removes that stall by handing the record to a background worker through a process-safe queue. This guide is a focused task within the Loguru Configuration and Sinks reference, part of the Modern Python Logging Libraries Deep Dive guide. For custom destinations that benefit from non-blocking dispatch, pair this with implementing custom sinks in Loguru.

Prerequisites

Install Loguru with a pinned range. The queue machinery uses only the standard library, so no extra dependency is needed.

pip install "loguru>=0.7.0,<0.8.0"

No environment variables are required. The queue lives in process memory; on fork-based multiprocessing the child inherits the same writer, and on spawn you re-add the sink in each process.

Implementation

enqueue=True is a single argument on logger.add, but using it correctly means understanding the worker/queue model, exception forwarding, and shutdown.

1. Understand the worker/queue model. When you add a sink with enqueue=True, Loguru does three things: it creates one multiprocessing.SimpleQueue per sink, spawns a daemon thread that loops on queue.get(), and replaces the synchronous write path with a queue.put(record). The hot path on the caller is therefore a pickle of the record plus a queue put — bounded, predictable work — while the unbounded part (formatting, encoding, the actual write/flush, lock acquisition) happens on the worker. Because there is exactly one worker per sink draining one queue, writes are serialized: two threads can never interleave bytes into the same file. The queue is unbounded, so back-pressure is your responsibility (see the slow-sink mistake below).

One consequence of the pickle step is worth internalizing: anything you attach to a record through logger.bind(...) or pass as a keyword must be picklable, because the record crosses the queue boundary before it is formatted. A bound database connection or an open socket will raise at put time. Keep enqueued records to plain data — strings, numbers, dicts — and resolve rich objects to their string form at the call site. The worker thread is a daemon, so it does not keep the interpreter alive on its own; that is precisely why an explicit drain on shutdown is mandatory rather than optional.

2. Add a sink with enqueue=True. Every logger.info(...) call serializes the record, pushes it onto the queue, and returns immediately; the worker formats and writes it.

from loguru import logger
import sys

logger.remove()                               # drop the default stderr sink
# enqueue=True offloads writes to a background worker thread
logger.add(sys.stderr, enqueue=True, level="INFO")
# A file sink benefits the most because disk writes block the caller
logger.add("app.log", enqueue=True, serialize=True, level="DEBUG")

logger.info("non-blocking write", request_id="req-9931")

Expected Output:

2026-06-19 10:15:30.123 | INFO     | __main__:<module>:9 - non-blocking write

3. Forward worker exceptions. A failure inside the background worker is invisible to the caller — the exception is raised on the worker thread, not at the call site. Keep catch=True (the default) so the worker prints a traceback to stderr instead of dying silently and freezing the queue. A dead worker is the worst failure mode: puts keep succeeding, the queue grows without bound, and nothing is ever written.

# catch=True (default) reports sink errors from the worker thread
logger.add("app.log", enqueue=True, catch=True)

4. Drain the queue on shutdown. Because records are buffered, a process that exits abruptly loses anything still queued. logger.complete() blocks until the worker has flushed every pending record across all enqueued sinks; logger.remove() drains, stops the worker thread, and joins it for that sink, closing the file. A clean shutdown calls complete() to flush and then remove() to tear the workers down.

from loguru import logger

logger.add("app.log", enqueue=True)
logger.info("important event")

logger.complete()                             # wait for the queue to drain
logger.remove()                               # join worker, flush remaining records

5. Use it from asyncio. A coroutine calling logger.info(...) with an enqueued sink never blocks the event loop, because the write happens on the worker thread. This is the main reason to reach for enqueue in an async service: even a fast local file write involves a syscall that, without enqueue, runs synchronously on the loop thread and adds tail latency to every awaiting request. With enqueue, the loop thread only pays for a queue put. The important nuance is shutdown: logger.complete() is awaitable, and from a coroutine you await logger.complete(), which yields control instead of busy-waiting until the queue drains. This matters because an asyncio program that calls the blocking form would stall the loop, defeating the purpose. Put the await logger.complete() in your application's shutdown hook (for example an ASGI lifespan shutdown) so in-flight log records are flushed before the loop closes. Non-blocking dispatch like this is the same goal pursued by stdlib's non-blocking logging with QueueHandler.

import asyncio
from loguru import logger

logger.remove()
logger.add("app.log", enqueue=True, serialize=True)


async def handler(n: int) -> None:
    logger.info("handled", n=n)               # returns instantly, no loop stall


async def main() -> None:
    await asyncio.gather(*(handler(i) for i in range(1000)))
    await logger.complete()                    # await the flush, then exit cleanly


asyncio.run(main())

Expected Output:

{"text": "...handled\n", "record": {"extra": {"n": 999}, "level": {"name": "INFO", "no": 20}, "message": "handled"}}

How enqueue makes logging multiprocess-safe

When multiple processes write to the same file directly, their writes interleave and produce corrupt or partial lines. With enqueue=True, the queue is a multiprocessing.SimpleQueue: child processes that inherit the logger push records onto the same queue, and exactly one worker in the parent dequeues and writes. Serialization to one writer is what makes the file safe.

The start method decides whether inheritance happens at all. Under fork (the default on Linux), the child is a copy-on-write clone of the parent, so it inherits the live queue object and the worker keeps running in the parent — children only ever put, the parent's worker writes. Under spawn (the default on macOS and Windows, and increasingly recommended on Linux to avoid fork-after-thread hazards), the child is a brand-new interpreter that re-imports your module; it does not inherit the parent's in-memory queue or worker thread, so a sink added only at parent import time does not exist in the child and its records vanish. The fix under spawn is to re-add the enqueued sink inside each child's startup. This single-writer model is the same correctness guarantee behind thread-safe logging in multiprocessing with the standard library.

import multiprocessing as mp
from loguru import logger

logger.add("workers.log", enqueue=True, serialize=True)   # one writer in the parent


def task(i: int) -> None:
    logger.info("child work", worker=i)        # safe: pushed to the shared queue


if __name__ == "__main__":
    mp.set_start_method("fork")                # inherit the parent's queue + worker
    procs = [mp.Process(target=task, args=(i,)) for i in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    logger.complete()                          # drain queued child records

Configuration options

Related shutdown calls: logger.complete() blocks (or awaits) until all enqueued records are written, and logger.remove(sink_id) drains then joins the worker for one sink.

Verification

Prove the buffering and drain behavior: write a batch, then call logger.complete() and confirm every line landed in the file. Counting lines after the flush should equal the number of records emitted.

from loguru import logger

logger.remove()
sink_id = logger.add("verify.log", enqueue=True, level="INFO")

for i in range(500):
    logger.info("event", i=i)

logger.complete()                              # ensure the queue is fully drained
logger.remove(sink_id)                         # join the worker

with open("verify.log", encoding="utf-8") as fh:
    lines = fh.readlines()
assert len(lines) == 500, f"lost records: got {len(lines)}"
print("all records flushed:", len(lines))

Expected Output:

all records flushed: 500

If the count is short, the process exited before the worker drained the queue, meaning a missing logger.complete() or logger.remove().

Common mistakes

Exiting before the queue drains. Buffered records are lost when the interpreter shuts down mid-flush. Always call logger.complete() (or await it from asyncio) and logger.remove() before the process terminates, especially in short-lived workers and CLI tools.

Relying on enqueue alone to handle slow network sinks. enqueue prevents the caller from blocking, but the queue is unbounded, so a slow custom sink lets the backlog grow in memory until the process is OOM-killed. Combine enqueue with a bounded internal buffer and a drop policy as shown in implementing custom sinks in Loguru.

Forgetting to re-add the sink under spawn-based multiprocessing. With the spawn start method, child processes do not inherit the parent's enqueued sink, so their records silently vanish. Re-run logger.add(..., enqueue=True) in each child's entry point, or use fork where inheritance applies.

Setting catch=False on an enqueued sink. Turning off catch means a single malformed record or a transient sink error raises on the worker thread and kills it. Once the worker is dead, every later logger.info still succeeds (the put works) but nothing is ever written, and the failure is silent. Leave catch=True so the worker logs the traceback and keeps draining.

Frequently Asked Questions