Logging in Python Runtimes and Frameworks

A logging configuration that behaves perfectly in a script can duplicate every line under Gunicorn, ignore the access log under Uvicorn, lose records after a fork, fight with a serverless runtime's own handler, or — when shipped inside a library — override the application that imports it. The logging module itself behaves the same everywhere; what differs is who else touches it and when. This guide covers the common execution environments and what each does to Python logging. It is part of the modern Python logging libraries deep dive section, and its child pages go deeper on Gunicorn and Uvicorn worker logging, logging in AWS Lambda Python handlers and logging for Python library authors.

Prerequisites

pip install "gunicorn>=22.0.0,<24.0.0" \
            "uvicorn>=0.30.0,<1.0.0" \
            "python-json-logger>=2.0.7,<4.0.0"

Concept and architecture

The standard library's logging is a single, process-global tree of loggers with handlers attached. Anything running in the process can add handlers, change levels or install a configuration, and whoever runs last wins — or, more often, everybody's handlers coexist and records are emitted several times.

Three properties of that design explain almost every runtime-specific problem.

Configuration is additive by default. basicConfig does nothing if the root already has handlers, unless forced. addHandler adds without removing. dictConfig replaces handlers on the loggers it names but leaves others alone. An application that adds its handler to a root logger that a runtime has already configured ends up with two handlers and every record twice.

Propagation is separate from handling. A logger with its own handlers still passes records to its parent unless propagation is disabled. Servers commonly give their own loggers dedicated handlers and disable propagation, which is why their records never reach the application's root handler and never get its formatter.

Handlers can own threads and file descriptors. A QueueListener runs a thread; a file handler holds a descriptor; a network handler holds a socket. None of those behave well across a fork: threads do not exist in the child, and shared descriptors are written by several processes at once. Configuration that creates such handlers must run in each process that will use them.

A fourth property is less obvious and matters in long-running workers: the logger registry never shrinks. Every call to getLogger with a new name creates a logger that lives for the rest of the process. Runtimes and frameworks create a bounded set, but application code that names loggers dynamically — per tenant, per request, per job — grows the registry without limit, and each new logger pays for a level lookup the first time it is used. Fixed, module-level logger names avoid this entirely.

Every environment in this guide is some combination of those three properties: something configured logging before you (additive), something configured its own loggers not to propagate (propagation), or something forked your process after configuration (resources).

Step-by-step implementation

Step 1 — Find out who configures logging first. Before writing any configuration, inspect the logging tree at the moment your code first runs. The output names every handler already attached and every logger with non-default settings, which is exactly the set of things your configuration must account for.

import logging

def describe_logging() -> None:
    root = logging.getLogger()
    print("root", logging.getLevelName(root.level), [type(h).__name__ for h in root.handlers])
    for name, lg in sorted(logging.Logger.manager.loggerDict.items()):
        if isinstance(lg, logging.Logger) and (lg.handlers or not lg.propagate or lg.level):
            print(f"  {name:28s} level={logging.getLevelName(lg.level):8s} "
                  f"propagate={lg.propagate} handlers={[type(h).__name__ for h in lg.handlers]}")

describe_logging()

Expected Output: under Uvicorn, the server's loggers are already configured, with their own handlers and propagation disabled.

root WARNING []
  uvicorn                      level=INFO     propagate=False handlers=['StreamHandler']
  uvicorn.access               level=INFO     propagate=False handlers=['StreamHandler']
  uvicorn.error                level=INFO     propagate=True handlers=[]

Step 2 — Take control of the root logger once. Apply one configuration at the earliest point your code runs, and replace handlers rather than adding to them. dictConfig with disable_existing_loggers: False keeps library loggers working; force=True on basicConfig removes existing root handlers first.

Step 3 — Bring the runtime's loggers under your handler. Name the server's loggers in your configuration, remove their handlers and let them propagate — or give them your handler explicitly. Either way, their records then pass through your formatter and match the rest of your output. Leaving them alone is how services end up with JSON application logs and plain-text access logs in the same stream.

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {"json": {"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
                            "fmt": "%(asctime)s %(levelname)s %(name)s %(message)s"}},
    "handlers": {"stdout": {"class": "logging.StreamHandler",
                            "stream": "ext://sys.stdout", "formatter": "json"}},
    "loggers": {
        # the server's loggers: no handlers of their own, records flow to root
        "uvicorn":        {"handlers": [], "propagate": True, "level": "INFO"},
        "uvicorn.access": {"handlers": [], "propagate": True, "level": "INFO"},
        "uvicorn.error":  {"handlers": [], "propagate": True, "level": "INFO"},
        "gunicorn.error": {"handlers": [], "propagate": True, "level": "INFO"},
        "gunicorn.access":{"handlers": [], "propagate": True, "level": "INFO"},
    },
    "root": {"handlers": ["stdout"], "level": "INFO"},
}

Step 4 — Initialise per process where the runtime forks. Anything that owns a thread or a connection — a queue listener, an OpenTelemetry log processor, a network handler — must be created in each worker after the fork. Prefork servers provide a hook for exactly this; using it is the difference between logging that works and a queue that fills and is never drained.

# gunicorn.conf.py
def post_fork(server, worker):
    from myservice.logsetup import start_queue_listener
    start_queue_listener()          # a thread that exists in THIS process

Step 5 — Never configure logging from library code. A library cannot know where the application wants records to go, and installing a handler makes that decision for every application that imports it. Libraries create named loggers, add a NullHandler at most, and leave everything else to the application.

Step 6 — Verify the final tree, in the real runtime. Run the inspection from step 1 again after configuration, under the real server, and confirm there is exactly one handler path for every logger that matters. Duplication and bypass are both visible in that output, and running it under a development server is not enough, because development servers often configure logging differently from their production counterparts.

Configuration reference

Async and concurrency considerations

The runtime determines the concurrency model, and the concurrency model determines which logging arrangements are safe.

Under a prefork server with synchronous workers, each worker is a single-threaded process. A StreamHandler writing to standard output is safe, because each process has its own buffer and writes below the pipe atomicity limit do not interleave. A shared file handler is not, because several processes append to the same file without coordination, which is the subject of thread-safe logging in multiprocessing.

Under an ASGI server, a single process runs an event loop, and every synchronous handler runs on the loop thread. A slow sink stalls every request, which is why the queue handler matters most here — and why the listener must be started inside the worker process, after any fork, as in step 4.

Under a threaded server, handlers are called from many threads, and the logging module's per-handler lock serialises them. A slow handler becomes a contention point across all request threads. Again the remedy is a queue, and again the listener must live in the process that serves requests.

Serverless runtimes add one more wrinkle: the process is frozen between invocations, so a queue listener's thread does not run while the function is idle. Records enqueued at the end of one invocation are written only when the next invocation thaws the process — which may be never. For serverless, a synchronous handler to standard output is usually the right choice, because the runtime captures standard output and the volume per invocation is small; this is developed in logging in AWS Lambda Python handlers.

Production code examples

A single entry point that configures logging correctly for whichever runtime is in use, detecting the ones that need special handling:

# logsetup.py
import logging
import logging.config
import os
import queue
from logging.handlers import QueueHandler, QueueListener

from pythonjsonlogger import jsonlogger

SERVER_LOGGERS = ("uvicorn", "uvicorn.access", "uvicorn.error",
                  "gunicorn", "gunicorn.access", "gunicorn.error",
                  "celery", "celery.task")
_listener: QueueListener | None = None


def _json_handler() -> logging.Handler:
    h = logging.StreamHandler()
    h.setFormatter(jsonlogger.JsonFormatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
    return h


def configure(use_queue: bool | None = None) -> None:
    serverless = "AWS_LAMBDA_FUNCTION_NAME" in os.environ
    if use_queue is None:
        use_queue = not serverless              # 1. frozen processes cannot drain a queue

    root = logging.getLogger()
    for h in list(root.handlers):               # 2. replace, never add
        root.removeHandler(h)
    root.setLevel(os.environ.get("LOG_LEVEL", "INFO"))
    root.addHandler(_json_handler())

    for name in SERVER_LOGGERS:                 # 3. server records through our formatter
        lg = logging.getLogger(name)
        lg.handlers.clear()
        lg.propagate = True


def start_queue_listener() -> None:
    """Call once per process, AFTER any fork."""
    global _listener
    root = logging.getLogger()
    target = root.handlers[0]
    q: "queue.Queue[logging.LogRecord]" = queue.Queue(maxsize=10_000)
    _listener = QueueListener(q, target, respect_handler_level=True)
    _listener.start()
    root.handlers = [QueueHandler(q)]

Expected Output: the tree after configuration, with one handler at the root and every server logger propagating to it.

root INFO ['QueueHandler']
  uvicorn                      level=NOTSET   propagate=True handlers=[]
  uvicorn.access               level=NOTSET   propagate=True handlers=[]
  uvicorn.error                level=NOTSET   propagate=True handlers=[]

Frameworks inside the runtime

The web framework adds a second layer on top of the server, and its logging behaviour matters too.

Django configures logging from the LOGGING setting at startup, applying it with dictConfig. Its default configuration adds handlers to the django logger that behave differently depending on DEBUG. Providing a complete LOGGING setting — with disable_existing_loggers: False and explicit entries for the django loggers — gives the application full control, as covered in logging configuration in Django settings.

Flask installs a default handler on the application's logger only if no handlers are configured when the logger is first accessed. Configuring logging before creating the application prevents it, and is simpler than removing it afterwards.

FastAPI and Starlette do not configure logging themselves; their output goes through whatever the ASGI server configured. The server's configuration is therefore the one to control, as described in configuring logging for FastAPI and Uvicorn.

Celery replaces the root logger's configuration when a worker starts, unless told not to. Setting worker_hijack_root_logger to false and configuring logging in the setup_logging signal handler keeps the application's configuration intact inside workers.

The common thread is that every framework has a documented way to say "logging is configured, leave it alone", and using it is always better than fighting the default configuration after it has been applied.

Choosing a library in each runtime

The runtime also affects which logging library fits, because the libraries differ in how they interact with the global logging tree that runtimes configure.

The standard library is what every runtime and framework expects. Servers, frameworks and third-party packages all log through it, so a service that also uses it has one tree to configure and one formatter to apply. Its weakness is ergonomics for structured data, which a JSON formatter and a consistent use of extra largely address.

structlog can either render records itself or pass them to the standard library for output. In any runtime where other components log through the standard library — which is all of them — the second arrangement is strongly preferable: structlog builds the event dictionary, and the standard library's handlers write it alongside the server's and libraries' records with one formatter. Configuring structlog to print directly means its records and everybody else's follow different paths, which reintroduces exactly the mixed-format problem this guide is about. The routing is covered in structlog architecture and setup.

Loguru replaces the standard library's handler model with its own sinks, and records from other libraries do not reach those sinks unless the standard library is intercepted and redirected. In runtimes with busy standard library loggers — servers, ORMs, HTTP clients — that interception is essential, and it is covered in intercepting standard logging with Loguru. Loguru's own process-safety features also interact with prefork servers, which is the subject of Loguru in multiprocessing workers.

Whichever library an application uses, the principle is the same: every record in the process, from every component, should reach one output path with one formatter. The runtime's own loggers are part of "every component", and they are the ones most often forgotten.

Checking a running service

Configuration mistakes in this area are easy to detect from outside, which makes a quick audit of a running service worthwhile whenever a runtime or framework changes.

Look for mixed formats in the output. A short sample of a service's standard output, checked for lines that do not parse as JSON, finds records bypassing the formatter immediately. Each such line has a logger name — or, for truly raw output, a recognisable shape — that identifies the component responsible.

Look for exact duplicates. Two identical records with the same timestamp to the microsecond are the signature of two handlers on the same path. Counting duplicates in a sample of a few thousand lines is enough to detect it.

Look for silence from components that should be logging. A service whose access log has no lines, or whose workers produce nothing after startup, has records going nowhere — usually a non-propagating logger with no handler, or a queue with no listener in the worker process.

Each check is a few lines of script against a sample of output, and running all three after any change to the server, framework or logging configuration catches the problems in this guide before they reach the log store.

Common mistakes

Adding a handler to an already-configured root. Error signature: every line appears twice. Root cause: the runtime's handler and yours both attached. Remediation: remove existing root handlers before adding yours.

Leaving server loggers alone. Error signature: plain-text access lines in an otherwise JSON stream, and parser failures in the collector. Root cause: the server's loggers have their own handlers and do not propagate. Remediation: clear their handlers and let them propagate.

A queue listener started before fork. Error signature: workers that log nothing, with no error. Root cause: the listener thread exists only in the parent. Remediation: start it in a post-fork hook.

Configuring logging in a library. Error signature: an application's output format changing when a dependency is upgraded. Root cause: the library installed a handler. Remediation: libraries use NullHandler only.

A queue in a serverless function. Error signature: the final records of an invocation missing or appearing in the next one. Root cause: the listener thread frozen between invocations. Remediation: a synchronous handler to standard output.

Dynamically named loggers. Error signature: memory growing slowly in a long-running worker, with the logging manager's registry holding thousands of entries. Root cause: a logger created per request or per tenant. Remediation: module-level loggers, with the variable part carried as a field.

Framework defaults re-applied. Error signature: a correct configuration that is overwritten a moment later. Root cause: the framework configures logging after your code did. Remediation: use the framework's documented switch to leave logging alone.

Frequently Asked Questions