Non-Blocking Logging with QueueHandler and QueueListener

When a logger writes directly to a file or network sink, the thread or event loop that called log.info pays the full I/O latency, which destroys tail latency under load. This walkthrough is for backend engineers and SREs who need logging to cost microseconds on the request path instead of milliseconds, by turning every emit into an in-memory enqueue and pushing the slow writes onto a background thread with QueueHandler and QueueListener. It sits inside the handler architecture guide, part of the Python Logging Fundamentals and Structured Data reference.

Prerequisites

# Standard library only. Verified on CPython 3.10-3.13; 3.12.x used throughout.
python --version   # CPython 3.8+ for the code, 3.12+ for the dictConfig wiring below

No third-party packages and no environment variables are required. Everything here uses logging, logging.handlers, and queue from the standard library. You should already have a working handler and formatter setup — if the sinks themselves are still undecided, start from the handler architecture guide and come back once you know which handlers the listener will own.

How QueueHandler and QueueListener work internally

QueueHandler is a deliberately thin handler. Its emit method does two cheap things: it calls self.prepare(record), then self.enqueue(record). The prepare step is the subtle part — it formats the message and, in recent CPython, detaches args, exc_info, and exc_text so that a stale or unpicklable argument cannot blow up later on the listener thread. After prepare, the record is a self-contained, safe-to-move object, and enqueue simply calls self.queue.put_nowait(record). No formatting of the final output, no file write, and no socket call happens on the producer.

QueueListener is the consumer half. Its start method spawns a single non-daemon thread running a _monitor loop that blocks on self.queue.get(), hands each record to self.handle(record), and calls self.queue.task_done(). handle walks the listener's owned handlers and calls each one's handle method, which is where the real StreamHandler or RotatingFileHandler finally does its blocking I/O — on the background thread, off the hot path. Because only one listener thread drains the queue, the owned handlers are never called concurrently, which sidesteps a whole class of interleaving bugs that plague directly shared file handlers.

The single-consumer design also gives you a free ordering guarantee: records are written to the sinks in the exact order they were enqueued, even when dozens of producer threads raced to enqueue them. This is why a queue-fronted file is more trustworthy during an incident than a file handler shared directly across threads, where per-handler locking still admits surprising interleaving at flush boundaries. The trade-off is that the listener is a single point of throughput: if your aggregate log rate exceeds what one thread can serialize and write, the queue backs up and the drop policy below starts shedding load. In practice one thread comfortably absorbs tens of thousands of records per second to a local file, and the bottleneck only appears with synchronous network sinks — which is itself an argument for exporting through a batching collector, the same reasoning behind the exporter design in distributed tracing with OpenTelemetry.

Implementation

Step 1 — Create a bounded queue. An unbounded queue lets memory grow without limit when the downstream sink stalls. A maxsize caps that growth and lets you choose what happens when it fills. Size it from measurement: peak records per second multiplied by the longest sink stall you intend to survive, at roughly one to two kilobytes of resident memory per prepared record.

import queue

# 10k records ~= 10-20 MB resident; absorbs a ~2s stall at 5k records/sec.
log_queue: "queue.Queue" = queue.Queue(maxsize=10_000)

Step 2 — Attach a QueueHandler as the only handler. The QueueHandler.emit method calls put_nowait, so emitting a record is a fast in-memory operation. Make it the sole handler on the application logger so no synchronous sink is reachable from the hot path. Set the level on the logger, not on the queue handler, so filtering happens before the enqueue and rejected records never occupy queue slots.

import logging
from logging.handlers import QueueHandler

queue_handler = QueueHandler(log_queue)

app_log = logging.getLogger("app")
app_log.setLevel(logging.DEBUG)
app_log.addHandler(queue_handler)
app_log.propagate = False        # avoid duplicate emission via the root logger

Step 3 — Run a QueueListener over the real handlers. The listener owns the slow handlers and drains the queue from a single background thread. respect_handler_level=True makes each downstream handler apply its own level, so a console handler can stay at INFO while a file handler captures DEBUG. This matters because, without it, the listener ignores per-handler levels entirely and every owned handler sees every record that cleared the logger's level — the console would then print the DEBUG lines you meant to keep on disk only.

import sys
import logging.handlers
from logging.handlers import QueueListener

console = logging.StreamHandler(sys.stdout)
console.setLevel(logging.INFO)
console.setFormatter(logging.Formatter("%(asctime)s %(levelname)-8s %(name)s | %(message)s"))

file_handler = logging.handlers.RotatingFileHandler(
    "app.log", maxBytes=10_485_760, backupCount=5
)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter(
    "%(asctime)s [%(levelname)s] %(name)s %(threadName)s - %(message)s"
))

listener = QueueListener(
    log_queue, console, file_handler, respect_handler_level=True
)
listener.start()                 # spawns the background draining thread

Putting a rotating handler behind the listener is not incidental — rollover performs a cascade of renames and a reopen, and doing that inline on a request thread is a measurable latency spike. The sizing and locking rules for that sink live in best practices for log rotation in Python.

Step 4 — Stop the listener on shutdown. listener.stop() enqueues a sentinel (an internal _sentinel, by default None), the _monitor loop drains everything ahead of the sentinel, sees it, exits, and stop joins the thread. The ordering guarantee is the point: every record enqueued before stop is processed before the thread terminates. Skipping stop lets the interpreter tear the thread down mid-drain and lose whatever was still buffered. Wire it into your shutdown path or an atexit hook.

import atexit

atexit.register(listener.stop)   # flush buffered records before the process exits

app_log.info("service started")
app_log.debug("warming caches")  # reaches the file, not the console

Declaring the same stack in dictConfig

From Python 3.12, logging.config.dictConfig understands queue handlers directly: give the handler a handlers list naming other configured handlers and the listener is constructed for you, with the queue created automatically if you do not supply one. Note the one sharp edge — dictConfig builds the listener but does not start it. This is the declarative equivalent of Steps 1 through 4 and slots into the pattern described in configuring logging with dictConfig.

import atexit
import logging
import logging.config

logging.config.dictConfig({
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "console": {"format": "%(asctime)s %(levelname)-8s %(name)s | %(message)s"},
    },
    "handlers": {
        "stdout": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "formatter": "console",
            "stream": "ext://sys.stdout",
        },
        "queue": {
            "class": "logging.handlers.QueueHandler",
            "handlers": ["stdout"],        # 3.12+: listener owns these
            "respect_handler_level": True,
        },
    },
    "loggers": {"app": {"level": "DEBUG", "handlers": ["queue"], "propagate": False}},
})

# dictConfig CREATES the listener but does not START it.
queue_handler = logging.getHandlerByName("queue")   # 3.12+
queue_handler.listener.start()
atexit.register(queue_handler.listener.stop)

Backpressure and what happens when the queue fills

By default QueueHandler blocks the caller when the queue is full, which reintroduces exactly the latency you were avoiding — and it does so at the worst possible moment, when the sink is already unhealthy and traffic is likely elevated. A production stack should decide explicitly what a full queue means: shed low-severity records, keep the ones that matter, and count what was lost so the gap is visible rather than silent.

import logging
import queue
from logging.handlers import QueueHandler


class DroppingQueueHandler(QueueHandler):
    """Drop records below ERROR when the queue is full instead of blocking."""

    def __init__(self, q: "queue.Queue") -> None:
        super().__init__(q)
        self.dropped = 0             # exported as a metric; never silently zero

    def enqueue(self, record: logging.LogRecord) -> None:
        try:
            self.queue.put_nowait(record)
        except queue.Full:
            if record.levelno >= logging.ERROR:
                try:
                    self.queue.put(record, timeout=0.25)   # bounded wait, not forever
                    return
                except queue.Full:
                    pass
            self.dropped += 1        # below ERROR, or the bounded wait expired

Two details make this safe. The wait for ERROR-and-above is bounded rather than indefinite, so a permanently wedged sink degrades into dropping instead of stalling every request thread behind it. And dropped is a real counter you should surface as a gauge or counter in your metrics pipeline, because a queue that sheds load without telling anyone produces log files that look complete and are not. The severity floor you choose here should match the contract described in log levels and severity mapping — if WARNING carries operational meaning in your service, raise the floor accordingly.

Threads, processes, and the fork hazard

queue.Queue is the right choice for the overwhelmingly common case: threads and asyncio coroutines inside a single process share memory, so a plain in-memory queue is fast and lossless. It does not cross process boundaries. When you run a pre-fork server such as Gunicorn or a multiprocessing worker pool and want every worker process to funnel records into one logging process, you need multiprocessing.Queue, which serializes each record and pipes it to the consumer. That serialization is exactly why QueueHandler.prepare strips unpicklable args and exc_info into a pre-formatted message — a record carrying a live socket or a lambda would otherwise fail to pickle on the way across.

There is a fork-specific hazard to plan for. If you create the queue and start the listener before forking workers (the typical Gunicorn lifecycle), each child inherits a copy of the in-memory queue.Queue, but the listener thread does not survive the fork — only the forking thread is carried into the child. The result is workers happily enqueuing into a queue that nothing drains: a slow memory leak that ends in an OOM kill. The robust pattern is to build the logging stack in a post-fork hook (post_fork in Gunicorn) so each process owns a live listener, or to switch to multiprocessing.Queue with the listener pinned to the parent. The same caution applies to file handles: two processes appending to one file without an external lock will interleave partial lines, which is the single-writer problem worked through in thread-safe logging in multiprocessing.

Under asyncio the story is simpler. QueueHandler.emit does not block the event loop because put_nowait returns immediately, so the same logger is safe to call from coroutines and threads alike, and the blocking writes happen only on the listener thread. The one thing the queue cannot do for you is carry request context: the listener thread has no access to the contextvars that were set on the producer, so any request identifier or trace identifier must be attached to the record at emit time — by a filter or a LoggerAdapter — as covered in adding trace IDs to log records. Formatting is likewise better handled by the listener's own formatters, using the JSON record shape from structured logging with the Python standard library.

Graceful shutdown ordering

Shutdown order is a correctness concern, not a cleanup detail. Stop the producers first — finish serving in-flight requests so no new records are created — then call listener.stop() so the queue drains to completion, and only then close the underlying handlers or flush an OTLP exporter. Reversing this drops the tail of your logs: if you close the file handler before the listener finishes, the listener's final writes hit a closed stream and raise on the background thread, where nothing is watching. With multiple listeners (for example, separate stdout and network pipelines), stop them in reverse order of how records flow so an upstream stage never feeds a stopped downstream one.

atexit is a reasonable backstop but a poor primary mechanism: it runs late, after framework shutdown hooks, and it will not save you if the process receives SIGKILL. Prefer an explicit hook in your server's lifespan or shutdown signal handler, and keep atexit.register(listener.stop) only as the last line of defence. logging.shutdown(), which the logging module registers with atexit itself, flushes and closes handlers attached to loggers — it knows nothing about your listener's owned handlers, which is precisely why the explicit stop is required.

Configuration options

Verification

Run the script from the implementation steps. The console (INFO and up) and the file (DEBUG and up) diverge, proving respect_handler_level works and that I/O happened off the producer thread.

Expected Output (stdout):

2026-06-19T12:18:44 INFO     app | service started

Expected Output (app.log):

2026-06-19T12:18:44 [INFO] app MainThread - service started
2026-06-19T12:18:44 [DEBUG] app MainThread - warming caches

The DEBUG line is absent from the console but present in the file, and both records flushed because listener.stop ran via the atexit hook before exit.

For the latency claim itself, measure it rather than trusting it. The following harness times the producer-side cost of an emit with a slow sink behind the queue, which is the number that matters for your request path.

import logging, queue, statistics, time
from logging.handlers import QueueHandler, QueueListener


class SlowHandler(logging.Handler):
    def emit(self, record):        # stands in for a stalled file or socket sink
        time.sleep(0.005)


q = queue.Queue(maxsize=10_000)
log = logging.getLogger("bench")
log.setLevel(logging.INFO)
log.addHandler(QueueHandler(q))
log.propagate = False
listener = QueueListener(q, SlowHandler())
listener.start()

samples = []
for _ in range(1000):
    t0 = time.perf_counter_ns()
    log.info("checkout completed order_id=%s", 41234)
    samples.append((time.perf_counter_ns() - t0) / 1000)   # microseconds
listener.stop()
print(f"p50={statistics.median(samples):.1f}us p99={sorted(samples)[989]:.1f}us")

Expected Output:

p50=3.4us p99=11.8us

Single-digit microseconds on the producer against a sink that takes 5 ms per record is the whole point: without the queue, the same loop would report a p50 near 5000 µs. Two further checks belong in a load test — watch log_queue.qsize() under peak traffic to confirm the bound is never approached in steady state, and assert that the dropped counter stays at zero outside of deliberate stall drills.

Common mistakes

Forgetting to stop the listener Error signature: the last few seconds of logs before a deploy or restart are simply missing, and the gap size grows with traffic. Root cause: records buffered in the queue at interpreter exit are lost when the listener thread is torn down mid-drain. Remediation: call listener.stop() in your shutdown hook, with atexit.register(listener.stop) as a backstop, so the sentinel flushes the queue.

Leaving the queue unbounded Error signature: RSS climbs steadily during a downstream sink outage and the container is OOM-killed with no application error to explain it. Root cause: maxsize=0 grows without limit, so a stalled sink converts a logging problem into a memory problem. Remediation: set an explicit bound and pair it with a drop policy that protects ERROR and CRITICAL and counts what it sheds.

Attaching the real handlers to the logger as well Error signature: every line appears twice in the console, and p99 latency never improves after adding the queue. Root cause: the console or file handler is still on the logger alongside the QueueHandler, so the synchronous write happens on the hot path and again on the listener thread. Remediation: remove all sink handlers from the logger; the listener must be the only owner of the real handlers.

Closing handlers before the listener drains Error signature: ValueError: I/O operation on closed file on a thread named Thread-1, or a truncated tail in the log file after shutdown. Root cause: handler.close() or an exporter shutdown ran before listener.stop(), so the listener's final writes landed on a closed sink. Remediation: stop producers, then stop the listener, then close handlers — in that order.

Starting the listener before forking workers Error signature: worker memory grows monotonically and no records ever reach the file, while the parent process logs normally. Root cause: the listener thread does not survive fork(), so each child inherits a queue with no consumer. Remediation: build the queue and start the listener in a post-fork hook, or move to a multiprocessing.Queue with a listener in a dedicated logging process.

Related

• Handler architecture — the parent guide covering handler topology, levels, and backpressure across sinks.
• Best practices for log rotation in Python — the rotating sink this listener should own, and why rollover must stay off the request thread.
• Thread-safe logging in multiprocessing — the single-writer pattern for pre-fork servers and worker pools.
• Configuring logging with dictConfig — declaring the queue handler, its listener, and its formatters in one config block.
• Async and non-blocking logging with Loguru enqueue — the same queueing idea expressed as a single sink parameter.

Frequently Asked Questions