Changing Python Log Levels at Runtime

The information you need during an incident is DEBUG-level, and the service is running at INFO. Restarting to raise the verbosity loses the state you were trying to inspect. This page covers the three mechanisms for changing a level in a live process — an admin endpoint, a signal, a watched file — and the two rules that decide whether the change has any visible effect. It builds on log levels and severity mapping, part of the Python logging fundamentals and structured data section.

Prerequisites

pip install "fastapi>=0.115.0,<1.0.0"

export LOG_ADMIN_TOKEN="$(openssl rand -hex 16)"
export LOG_LEVEL_TTL=900          # seconds before a runtime override expires

Implementation

Step 1 — Change the level on the logger you mean. setLevel clears the effective-level cache for the whole hierarchy itself, so no extra step is needed. Name the narrowest logger that covers what you are debugging: myapp.orders.checkout, not myapp.

import logging

def set_level(name: str, level: str) -> dict:
    logger = logging.getLogger(name)
    logger.setLevel(getattr(logging, level.upper()))     # clears the cache for you
    return {"logger": name, "level": logging.getLevelName(logger.level)}

Step 2 — Make sure the handlers will accept the records. This is the failure everyone hits once. Put handlers at NOTSET so they take whatever the logger sends, and keep volume control in the logger levels — the ones you can change at runtime.

for handler in logging.getLogger().handlers:
    handler.setLevel(logging.NOTSET)                     # accept whatever arrives

Step 3 — Expose it behind an admin route with a TTL. A debug level enabled during an incident and forgotten is a permanent cost increase. Give every override an expiry and a background task that restores the previous value.

import asyncio
import os
from fastapi import APIRouter, Header, HTTPException

router = APIRouter(prefix="/admin/logging")
_ADMIN_TOKEN = os.environ["LOG_ADMIN_TOKEN"]

@router.post("/level")
async def change_level(logger: str, level: str, x_admin_token: str = Header(default="")):
    if x_admin_token != _ADMIN_TOKEN:                    # never leave this open
        raise HTTPException(status_code=403)
    target = logging.getLogger(logger)
    previous = target.level
    target.setLevel(getattr(logging, level.upper()))
    logging.getLogger("admin").warning(
        "log level changed", extra={"target": logger, "level": level.upper()},
    )
    asyncio.create_task(_restore(logger, previous))      # expiry, not a promise to remember
    return {"logger": logger, "level": level.upper(), "expires_in": int(os.environ.get("LOG_LEVEL_TTL", 900))}

async def _restore(name: str, level: int) -> None:
    await asyncio.sleep(int(os.environ.get("LOG_LEVEL_TTL", 900)))
    logging.getLogger(name).setLevel(level)
    logging.getLogger("admin").warning("log level restored", extra={"target": name})

The change itself is logged at WARNING. An override that alters what the service records must be visible in what the service records, or the next person reading the log has no way to know why the volume changed.

Step 4 — Add a signal path. When the HTTP surface is the thing that is broken, a signal still gets through. SIGUSR2 is unused by the interpreter and safe to claim.

import signal

_LEVELS = [logging.INFO, logging.DEBUG]

def _cycle(signum, frame) -> None:
    root = logging.getLogger()
    nxt = _LEVELS[(_LEVELS.index(root.level) + 1) % len(_LEVELS)] if root.level in _LEVELS else logging.DEBUG
    root.setLevel(nxt)
    logging.getLogger("admin").warning("root level cycled", extra={"level": logging.getLevelName(nxt)})

signal.signal(signal.SIGUSR2, _cycle)

kill -USR2 "$(pgrep -f 'uvicorn app:app' | head -1)"

Keep the handler tiny. Signal handlers run between bytecodes on the main thread, and anything that acquires a lock there can deadlock against whatever held it when the signal arrived — setLevel and one log call is about the safe limit.

Step 5 — Reach every worker. setLevel changes one interpreter. In a four-worker deployment, an HTTP call changes one quarter of the service. Where that matters, keep the desired levels in a small file that each worker re-reads on a timer.

import json, pathlib, threading, time

LEVELS_FILE = pathlib.Path("/etc/app/log-levels.json")

def _watch(interval: float = 15.0) -> None:
    last = None
    while True:
        try:
            stamp = LEVELS_FILE.stat().st_mtime
            if stamp != last:
                last = stamp
                for name, level in json.loads(LEVELS_FILE.read_text()).items():
                    logging.getLogger(name).setLevel(level)
        except FileNotFoundError:
            pass
        time.sleep(interval)

threading.Thread(target=_watch, name="log-level-watch", daemon=True).start()

Configuration options

Verification

curl -s -XPOST -H "X-Admin-Token: $LOG_ADMIN_TOKEN" \
  'localhost:8000/admin/logging/level?logger=app.orders&level=DEBUG'

Expected Output:

{"logger": "app.orders", "level": "DEBUG", "expires_in": 900}

{"levelname": "WARNING", "name": "admin", "message": "log level changed", "target": "app.orders", "level": "DEBUG"}
{"levelname": "DEBUG", "name": "app.orders.checkout", "message": "cart recalculated", "items": 3}

The second line is the proof: a DEBUG record from a child of the logger you changed, which confirms both that the effective level propagated down the hierarchy and that the handler accepted it. If the first line appears and the second never does, the handler is still at INFO.

Common mistakes

The level changes and the output does not

Error signature: the admin endpoint returns success, logger.getEffectiveLevel() reports 10, and no DEBUG records appear. Root cause: the handler is at INFO and discards them after the logger let them through. Remediation: set handlers to NOTSET and keep volume control in logger levels.

One worker in four gets the change

Error signature: DEBUG records appear intermittently, roughly a quarter of the time. Root cause: the HTTP call reached one worker process. Remediation: distribute through a watched file or a control-plane message, or address workers individually by process ID.

DEBUG is still on three weeks later

Error signature: log volume and storage costs stepped up on an incident date and never came back down. Root cause: a runtime override with no expiry. Remediation: give every override a TTL, log the change at WARNING, and alert when a non-default level has been in place for longer than the TTL allows.

Making the override safe to hand out

A runtime level change is a production control, and the difference between a useful one and a liability is entirely in the guardrails around it. Four are worth building before anyone needs them at three in the morning.

Bound the blast radius. Refuse to set the root logger to DEBUG. It is almost never what the person wants — they want one subsystem — and it is reliably what they type when the service is on fire. Accepting a logger name and rejecting the empty string costs one line and prevents the change that turns a latency incident into a log-volume incident on top of it.

FORBIDDEN = {"", "root"}

if logger in FORBIDDEN and level.upper() == "DEBUG":
    raise HTTPException(status_code=400, detail="refusing DEBUG on the root logger")

Bound the duration. Covered above; the point worth repeating is that the TTL must survive the process that set it. An asyncio.sleep in a task works until the worker restarts, at which point the level resets by accident and nobody notices either the reset or the original override. If the change matters enough to audit, the desired state belongs in a file or a key that each worker re-reads.

Record who and why. An override changes what the service records, so it belongs in the record. A WARNING carrying the caller's identity, the target logger, the new level and the expiry means the next person reading the log can tell why the shape changed. Without it, a volume step in a chart is unexplained forever.

Expose the current state. A read endpoint that lists non-default levels is the thing that finds the override somebody forgot. It is three lines and it turns "is anything still turned up?" from an investigation into a query.

@router.get("/level")
async def list_overrides():
    out = {}
    for name in sorted(logging.Logger.manager.loggerDict):
        log = logging.getLogger(name)
        if log.level != logging.NOTSET:                 # an explicit level was set
            out[name] = logging.getLevelName(log.level)
    return {"overrides": out}

The alternative worth considering first

Before building the endpoint, it is worth asking whether the level is the right control at all. The reason to raise a level is almost always "I want more detail about the requests that are failing", and levels answer that badly: they give more detail about every request, including the overwhelming majority that are fine.

Two mechanisms answer the actual question better. A MemoryHandler that flushes on error gives full DEBUG context for exactly the requests that failed, permanently, with no operator action and no volume cost for the rest. And trace sampling with a rule that always keeps traces containing an error gives the same property for the trace signal.

Runtime level changes remain worth having for the cases those do not cover: a subsystem that is misbehaving without erroring, a suspicion about a code path that succeeds but does the wrong thing, or a library whose DEBUG output is the only documentation of its behaviour. Those are real, and they are a smaller set than the endpoint's usage suggests.

Related

• Log levels and severity mapping — the parent guide: what each level should mean before you start changing them.
• How to configure Python logging for production — the static baseline these overrides sit on top of.
• Buffering log records with MemoryHandler — per-request detail without changing a level at all.
• Configuring logging with dictConfig — why re-running the whole config at runtime is riskier than one setLevel.
• Rate limiting and sampling noisy loggers — keeping a DEBUG session from flooding the pipeline.

Frequently Asked Questions