Async and Non-Blocking Logging with Loguru enqueue

A logging call that writes to a slow file, a network socket, or a sink under lock contention stalls the thread that made it, and in an asyncio service that stall lands on the event loop where every other request is waiting. This page is for backend engineers and SREs running Loguru in async or multiprocess Python services who need logging calls to return in constant time and still lose nothing at shutdown. It sits within the Loguru configuration and sinks reference, part of the Modern Python Logging Libraries Deep Dive guide, and it covers exactly one argument — enqueue=True — and everything that argument implies about pickling, worker failure, and drain ordering.

Prerequisites

Install Loguru with a pinned range. The queue machinery uses only the standard library, so no extra dependency is needed.

pip install "loguru>=0.7.0,<0.8.0"

No environment variables are required. The queue lives in process memory, so nothing has to be provisioned or reachable over the network; on fork-based multiprocessing the child inherits the same writer, and on spawn you re-add the sink in each process. If you are still deciding whether Loguru is the right base for the service at all, the trade-offs are laid out in choosing a logging library for FastAPI.

Implementation

enqueue=True is a single argument on logger.add, but using it correctly means understanding the worker/queue model, what may cross the queue, exception forwarding, and shutdown ordering.

1. Understand the worker/queue model. When you add a sink with enqueue=True, Loguru does three things: it creates one multiprocessing.SimpleQueue per sink, spawns a daemon thread that loops on queue.get(), and replaces the synchronous write path with a queue.put(record). The hot path on the caller is therefore a pickle of the record plus a queue put — bounded, predictable work — while the unbounded part (formatting, encoding, the actual write/flush, lock acquisition) happens on the worker. Because there is exactly one worker per sink draining one queue, writes are serialized: two threads can never interleave bytes into the same file. The queue is unbounded, so back-pressure is your responsibility (see the slow-sink mistake below).

2. Keep enqueued records picklable. Anything you attach to a record through logger.bind(...) or pass as a keyword must be picklable, because the record crosses the queue boundary before it is formatted. A bound database connection, an open socket, or a lambda raises PicklingError at put time — at the call site, not on the worker. Keep enqueued records to plain data (strings, numbers, dicts) and resolve rich objects to their string form where you log them. This is rarely a real constraint in practice, because the fields worth binding are correlation values such as request and trace identifiers, exactly the pattern described in adding trace IDs to log records.

3. Add a sink with enqueue=True. Every logger.info(...) call serializes the record, pushes it onto the queue, and returns immediately; the worker formats and writes it.

from loguru import logger
import sys

logger.remove()                               # drop the default stderr sink
# enqueue=True offloads writes to a background worker thread
logger.add(sys.stderr, enqueue=True, level="INFO")
# A file sink benefits the most because disk writes block the caller
logger.add("app.log", enqueue=True, serialize=True, level="DEBUG")

logger.info("non-blocking write", request_id="req-9931")

Expected Output:

2026-06-19 10:15:30.123 | INFO     | __main__:<module>:9 - non-blocking write

4. Forward worker exceptions. A failure inside the background worker is invisible to the caller — the exception is raised on the worker thread, not at the call site. Keep catch=True (the default) so the worker prints a traceback to stderr instead of dying silently and freezing the queue. A dead worker is the worst failure mode: puts keep succeeding, the queue grows without bound, and nothing is ever written.

# catch=True (default) reports sink errors from the worker thread
logger.add("app.log", enqueue=True, catch=True)

5. Drain the queue on shutdown. Because records are buffered, a process that exits abruptly loses anything still queued. logger.complete() blocks until the worker has flushed every pending record across all enqueued sinks; logger.remove() drains, stops the worker thread, and joins it for that sink, closing the file. The worker is a daemon thread, so it will not keep the interpreter alive on your behalf — an explicit drain is mandatory, not a nicety. A clean shutdown calls complete() to flush and then remove() to tear the workers down, in that order.

from loguru import logger

logger.add("app.log", enqueue=True)
logger.info("important event")

logger.complete()                             # wait for the queue to drain
logger.remove()                               # join worker, flush remaining records

6. Use it from asyncio. A coroutine calling logger.info(...) with an enqueued sink never blocks the event loop, because the write happens on the worker thread. This is the main reason to reach for enqueue in an async service: even a fast local file write involves a syscall that, without enqueue, runs synchronously on the loop thread and adds tail latency to every awaiting request. With enqueue, the loop thread only pays for a queue put. The nuance is shutdown: logger.complete() is awaitable, and from a coroutine you await logger.complete(), which yields control instead of stalling the loop until the queue drains. Put that await in your application's shutdown hook — an ASGI lifespan shutdown, for instance — so in-flight records are written before the loop closes. Non-blocking dispatch like this is the same goal pursued by the standard library's non-blocking logging with QueueHandler, and the same discipline that keeps blocking calls off the loop in async tracing patterns.

import asyncio
from loguru import logger

logger.remove()
logger.add("app.log", enqueue=True, serialize=True)


async def handler(n: int) -> None:
    logger.info("handled", n=n)               # returns instantly, no loop stall


async def main() -> None:
    await asyncio.gather(*(handler(i) for i in range(1000)))
    await logger.complete()                    # await the flush, then exit cleanly


asyncio.run(main())

Expected Output:

{"text": "...handled\n", "record": {"extra": {"n": 999}, "level": {"name": "INFO", "no": 20}, "message": "handled"}}

How enqueue makes logging multiprocess-safe

When multiple processes write to the same file directly, their writes interleave and produce corrupt or partial lines. With enqueue=True, the queue is a multiprocessing.SimpleQueue: child processes that inherit the logger push records onto the same queue, and exactly one worker in the parent dequeues and writes. Serialization to one writer is what makes the file safe, and it is the same single-writer guarantee behind thread-safe logging in multiprocessing with the standard library.

The start method decides whether inheritance happens at all. Under fork (the default on Linux), the child is a copy-on-write clone of the parent, so it inherits the live queue object while the worker keeps running in the parent — children only ever put, the parent's worker writes. Under spawn (the default on macOS and Windows, and increasingly recommended on Linux to avoid fork-after-thread hazards), the child is a brand-new interpreter that re-imports your module; it does not inherit the parent's in-memory queue or worker thread, so a sink added only at parent import time does not exist in the child and its records vanish. The fix under spawn is to re-add the enqueued sink inside each child's startup, guarded so it runs once per process.

Single-writer ownership also settles a question that trips teams up when they combine enqueue with rotation: because only the worker touches the file, only the worker rotates it, so there is no window in which two processes both rename and reopen the same path. That makes enqueue=True the prerequisite for safely applying the policies in best practices for log rotation in Python to a multiprocess service.

import multiprocessing as mp
from loguru import logger

logger.add("workers.log", enqueue=True, serialize=True)   # one writer in the parent


def task(i: int) -> None:
    logger.info("child work", worker=i)        # safe: pushed to the shared queue


if __name__ == "__main__":
    mp.set_start_method("fork")                # inherit the parent's queue + worker
    procs = [mp.Process(target=task, args=(i,)) for i in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    logger.complete()                          # drain queued child records

Expected Output (workers.log):

{"text": "...child work\n", "record": {"extra": {"worker": 0}, "process": {"id": 41182}, "message": "child work"}}
{"text": "...child work\n", "record": {"extra": {"worker": 1}, "process": {"id": 41183}, "message": "child work"}}

Each line is whole and carries its originating process.id, which is the observable proof that one writer serialized four producers.

Configuration options

Two shutdown calls belong with this table: logger.complete() blocks (or awaits) until all enqueued records are written, and logger.remove(sink_id) drains, then joins the worker for one sink and closes its file handle.

Verification

Prove both properties separately: that the queue really drains, and that the caller really does not wait for the sink.

For the drain, write a batch, call logger.complete(), and count lines. The count must equal the number of records emitted.

from loguru import logger

logger.remove()
sink_id = logger.add("verify.log", enqueue=True, level="INFO")

for i in range(500):
    logger.info("event", i=i)

logger.complete()                              # ensure the queue is fully drained
logger.remove(sink_id)                         # join the worker

with open("verify.log", encoding="utf-8") as fh:
    lines = fh.readlines()
assert len(lines) == 500, f"lost records: got {len(lines)}"
print("all records flushed:", len(lines))

Expected Output:

all records flushed: 500

If the count is short, the process exited before the worker drained the queue, meaning a missing logger.complete() or logger.remove().

For the non-blocking property, add a deliberately slow callable sink and time the call site. Without enqueue the caller absorbs the sleep; with it, the caller returns in microseconds while the worker absorbs the delay during complete().

import time
from loguru import logger

def slow_sink(message) -> None:
    time.sleep(0.05)                           # stand-in for a slow network write

logger.remove()
logger.add(slow_sink, enqueue=True)

start = time.perf_counter()
for _ in range(20):
    logger.info("queued")
caller = time.perf_counter() - start
logger.complete()                              # the 1s of sink time is paid here
print(f"caller spent {caller * 1000:.1f} ms for 20 records")

Expected Output:

caller spent 3.4 ms for 20 records

Twenty records against a 50 ms sink would cost the caller a full second synchronously; a few milliseconds is the signal that dispatch is genuinely off the calling thread.

Common mistakes

• Error signature: the last few seconds of logs are missing after every deploy or CLI run, with no error and a truncated final line. Root cause: the interpreter exited while records were still in the queue, and the daemon worker was torn down mid-flush. Remediation: call logger.complete() — or await logger.complete() from asyncio — followed by logger.remove() in the shutdown path, and register it with atexit for short-lived workers and CLI tools.

• Error signature: resident memory climbs steadily during a downstream outage until the process is OOM-killed, while logging calls stay fast. Root cause: enqueue keeps the caller non-blocking but the queue is unbounded, so a slow or stalled sink lets the backlog grow without limit. Remediation: put a bounded internal buffer and an explicit drop policy inside the sink itself, as shown in implementing custom sinks in Loguru, and alert on queue depth rather than on the disappearance of logs.

• Error signature: the log file contains records from the parent only; every child process logs nothing, and the code works on Linux but not on macOS. Root cause: the spawn start method gives the child a fresh interpreter that never inherited the parent's queue or worker, so no sink exists in the child. Remediation: re-run logger.add(..., enqueue=True) inside each child's entry point (or an initializer passed to the pool), or select fork explicitly where inheritance applies.

• Error signature: logging calls keep succeeding for hours but the sink stops receiving anything, and one traceback appeared on stderr right before the silence began. Root cause: catch=False on an enqueued sink let a single malformed record or transient sink error kill the worker thread; puts still succeed into a queue nobody drains. Remediation: leave catch=True so the worker logs the traceback and keeps draining, and treat "queue depth rising while write rate is zero" as a dead-worker alarm.

• Error signature: PicklingError: Can't pickle <class 'sqlite3.Connection'> raised from the logger.info call itself. Root cause: an unpicklable object was attached with logger.bind(...) or passed as a keyword, and the record is pickled before it crosses the queue. Remediation: bind plain data only, converting rich objects to identifiers or strings at the call site — the same contextual-binding discipline covered in binding context variables in structlog.

Related

• Loguru configuration and sinks — the parent reference covering sink topology, rotation, retention and structured output.
• Implementing custom sinks in Loguru — where to add the bounded buffer and drop policy that enqueue alone does not give you.
• Non-blocking logging with QueueHandler — the standard-library equivalent of this pattern, with an explicit listener you own.
• Thread-safe logging in multiprocessing — the single-writer model applied to logging handlers across processes.
• Loguru vs structlog for microservices — how the two libraries compare once structured output and async dispatch are both requirements.

Frequently Asked Questions