Exception and Traceback Logging in Python

An exception that reaches your log as ERROR: something went wrong costs an hour of an on-call engineer's night. This guide is for backend engineers and SREs who want tracebacks that arrive complete, grouped, and queryable — and who have been burned by a formatter that swallowed the stack, a queue that stripped it, or a message field that leaked a connection string. It is part of the Python logging fundamentals and structured data section, and it assumes you already have structured logging with the standard library in place.

Three mechanics explain almost every exception-logging bug in Python: the record carries a live (type, value, traceback) triple rather than a string, the first formatter to touch it caches the rendered result for everyone else, and anything that copies or pickles the record throws that triple away. Get those three straight and the rest — chained exceptions, async escape routes, frame limits, redaction — falls into place. The concrete walkthroughs live in logging exceptions and tracebacks, capturing unhandled exceptions and warnings, and redacting sensitive data in log records.

Prerequisites

Nothing beyond the standard library is required for capture; the optional pins below cover structured output and the OpenTelemetry correlation described later.

pip install "python-json-logger>=2.0.7,<4.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Set the two environment variables that change how much a traceback costs you at volume. The frame limit is enforced in your own formatter — Python has no global knob for it — so treat these as the values your formatter reads.

export LOG_TRACEBACK_FRAMES=20      # frames kept per exception, innermost first
export LOG_TRACEBACK_MAX_BYTES=8192 # hard cap on the serialised exception object

If your handlers already sit behind a queue, read non-blocking logging with QueueHandler first: the queue boundary is where tracebacks most often disappear, and the fix belongs in prepare() rather than in the formatter.

Concept and architecture

A Python log record does not contain a traceback. It contains exc_info, a three-element tuple of the exception class, the exception instance, and a traceback object — all live references into the interpreter's state. logging.Formatter.format() is what turns that into text, by calling formatException(), which by default defers to traceback.print_exception().

The consequence that catches people is the caching. Formatter.format() runs roughly this sequence: render the message, then, if record.exc_info is set and record.exc_text is not, call formatException() and store the result on the record. Records are shared objects — a logger hands the same record to every matching handler — so whichever handler runs first decides what every subsequent formatter sees. A JSON handler that emits a structured exception object and a file handler that wants a classic indented traceback cannot both get their way unless one of them clears the cache.

The second structural fact is that exc_info does not survive serialisation. Traceback objects cannot be pickled. Anything that needs to move a record between processes — QueueHandler with a multiprocessing.Queue, SocketHandler, a custom fan-out that deep-copies — must render the traceback to text first and drop the triple. QueueHandler.prepare() does exactly that, which is why a traceback arrives glued onto the end of the message with exc_info set to None on the other side.

The third is the chain. Since Python 3, every exception raised while another is being handled carries a link to it: __context__ when it happens implicitly, __cause__ when you write raise NewError from exc, and __suppress_context__ when you write from None. The default traceback renderer walks those links and prints all of them, joined by one of two sentences that mean quite different things.

Step-by-step implementation

Step 1 — Log the exception where you handle it, once. logger.exception() is logger.error() with exc_info=True; both attach the exception currently being handled. Call it at the boundary that decides what happens next — the request handler, the task wrapper, the retry loop — not at every frame on the way up. When you hold a specific exception object rather than the active one, pass it directly: exc_info accepts an exception instance as well as True.

import logging

logger = logging.getLogger(__name__)

def handle_order(payload: dict) -> None:
    try:
        process(payload)
    except ValueError as exc:                       # translate, keep the link
        raise OrderRejected("payload failed validation") from exc

def request_boundary(payload: dict) -> int:
    try:
        handle_order(payload)
    except OrderRejected:
        logger.exception("order rejected", extra={"order_id": payload.get("id")})
        return 400
    except Exception as exc:                        # unexpected: log the object we hold
        logger.error("order failed", exc_info=exc, extra={"order_id": payload.get("id")})
        return 500
    return 202

Both except arms produce a record carrying the full chain — OrderRejected with its __cause__ pointing at the original ValueError — because the chain lives on the exception object, not on the log call.

Step 2 — Turn the traceback into fields. The default rendering is one multi-line string, which a log backend stores as an opaque blob: you cannot facet by exception type, group by the raising module, or alert on a specific frame. Override formatException() to return a serialised object instead, and clear record.exc_text first so this formatter is not handed some other handler's rendering.

import json
import logging
import traceback

MAX_FRAMES = 20

def _serialise(exc: BaseException, depth: int = 0) -> dict:
    """One exception as fields; recurse into the chain, innermost link last."""
    frames = traceback.extract_tb(exc.__traceback__)[-MAX_FRAMES:]
    node = {
        "type": type(exc).__name__,
        "module": type(exc).__module__,
        "message": str(exc)[:512],
        "frames": [
            {"file": f.filename, "line": f.lineno, "func": f.name, "code": f.line}
            for f in frames
        ],
    }
    if depth < 3:                                   # bound the chain, not just the frames
        if exc.__cause__ is not None:
            node["cause"] = _serialise(exc.__cause__, depth + 1)
        elif exc.__context__ is not None and not exc.__suppress_context__:
            node["context"] = _serialise(exc.__context__, depth + 1)
    return node

class StructuredExceptionFormatter(logging.Formatter):
    def formatException(self, ei) -> str:
        return json.dumps(_serialise(ei[1]), default=str)

    def format(self, record: logging.LogRecord) -> str:
        record.exc_text = None                      # never inherit another handler's render
        payload = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        if record.exc_info:
            payload["exception"] = json.loads(self.formatException(record.exc_info))
        return json.dumps(payload, default=str)

Two details earn their keep. extract_tb(...)[-MAX_FRAMES:] keeps the innermost frames, which is where the error actually happened — slicing from the front gives you twenty frames of framework middleware and none of your code. And str(exc)[:512] bounds the message, because a ValidationError from a large payload will happily hand you fifty kilobytes of prose.

Step 3 — Distinguish the two chain relationships when you serialise. The code above records cause and context under different keys deliberately. A cause link is a design decision and belongs in the grouping key: alert on OrderRejected, and read the KeyError underneath as detail. A context link means a second exception fired while the first was being handled, which is nearly always its own bug — a finally block that raised, a rollback that failed. Giving them different key names lets you write that alert.

Step 4 — Bound the payload. A RecursionError carries a thousand frames; a deeply nested chain multiplies that. The frame slice caps one exception, depth < 3 caps the chain, and a final size check caps the whole thing so a single record cannot fill a bounded queue on its own.

def _bounded(payload: dict, limit: int = 8192) -> str:
    body = json.dumps(payload, default=str)
    if len(body) <= limit:
        return body
    exc = payload.get("exception", {})
    exc["frames"] = exc.get("frames", [])[-3:]      # keep the innermost three
    exc["truncated"] = True
    return json.dumps(payload, default=str)[:limit]

Step 5 — Close the escape hatches. Everything above assumes an except block ran. Exceptions that never reach one — a thread that dies, a task nobody awaited, a SystemExit in a worker — leave nothing in the log unless you install the interpreter's hooks. That plumbing is the subject of capturing unhandled exceptions and warnings; the short version is four hooks, and each one covers a different escape route.

Step 6 — Redact on the producing thread. Exception messages carry whatever was in the failing call: a DSN with a password, a token in a URL, a row of customer data. Attach the redaction filter to the logger rather than to a handler, so it runs before the record is queued, copied, or shipped. The pattern set and its failure modes are in redacting sensitive data in log records.

Configuration reference

Async and concurrency considerations

contextvars and asyncio change where an exception surfaces, not how it is formatted — and "where" is what decides whether anything logs it at all.

An exception inside a coroutine you await propagates normally into your except block. An exception inside a task you created and never awaited does not: it is stored on the task, and only when that task is garbage-collected does asyncio complain, through the loop's exception handler, often long after the request finished and with no request context left in contextvars. Set a loop exception handler at startup and the record at least reaches your sinks:

import asyncio
import logging

logger = logging.getLogger("asyncio.unhandled")

def _handler(loop, context):
    exc = context.get("exception")
    logger.error(context.get("message", "unhandled asyncio error"),
                 exc_info=exc if exc else None,
                 extra={"future": repr(context.get("future"))})

asyncio.get_event_loop().set_exception_handler(_handler)

Threads are separate again: a Thread whose target raises does not touch sys.excepthook, it goes to threading.excepthook. Worker processes started by multiprocessing inherit neither hook reliably on the spawn start method, so the child must install them itself in its initialiser. The matrix below is the one worth pinning above the desk.

The queue boundary deserves its own warning. QueueHandler.prepare() calls self.format(record), assigns the result to record.msg, and then sets exc_info, exc_text, args and stack_info to None so the object can be pickled onto a multiprocessing.Queue. Downstream formatters therefore see a message with a traceback already stuck to it and no exception to format. If your queue is an in-process queue.Queue, nothing is ever pickled and you can keep the live triple:

from logging.handlers import QueueHandler

class LiveQueueHandler(QueueHandler):
    """In-process queue: nothing is pickled, so keep exc_info intact."""
    def prepare(self, record):
        return record                                # no format, no copy, no strip

That is safe only when producer and consumer share a heap. Across processes, keep the default and let prepare() render — then make sure the producing side owns the formatter you actually want, because that is the render that survives.

Production code examples

A complete service-side setup: structured exceptions, redaction on the producing thread, trace correlation, and the four hooks. The trace fields come from the active span, which is what makes an error in the log clickable through to the trace described in adding trace IDs to log records.

# observability/errors.py — import once at startup
import logging
import logging.config
import re
import sys
import threading
import warnings

SECRET = re.compile(
    r"(password|passwd|token|api[_-]?key|secret|authorization)\s*[=:]\s*[^\s,;)'\"]+",
    re.I,
)

class RedactFilter(logging.Filter):
    """Runs on the producing thread, before any queue or sink sees the record."""
    def filter(self, record: logging.LogRecord) -> bool:
        if isinstance(record.msg, str):
            record.msg = SECRET.sub(r"\1=***", record.msg)
        if record.exc_info and record.exc_info[1] is not None:
            exc = record.exc_info[1]
            if exc.args and isinstance(exc.args[0], str):
                exc.args = (SECRET.sub(r"\1=***", exc.args[0]),) + exc.args[1:]
        return True

def install_hooks(logger: logging.Logger) -> None:
    def _sys_hook(exc_type, exc, tb):
        if issubclass(exc_type, KeyboardInterrupt):      # let Ctrl-C stay quiet
            sys.__excepthook__(exc_type, exc, tb)
            return
        logger.critical("uncaught exception", exc_info=(exc_type, exc, tb))

    def _thread_hook(args):
        logger.critical("uncaught exception in thread %s", args.thread.name,
                        exc_info=(args.exc_type, args.exc_value, args.exc_traceback))

    sys.excepthook = _sys_hook
    threading.excepthook = _thread_hook
    logging.captureWarnings(True)                        # DeprecationWarning -> py.warnings
    warnings.simplefilter("default")
    logging.raiseExceptions = False                      # a broken sink must not spam stderr

Wire it with dictConfig so the filter is attached at the logger, above the handlers — the placement that guarantees redaction happens before the record is queued. The full-graph version of this configuration is in configuring logging with dictConfig.

CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "filters": {"redact": {"()": "observability.errors.RedactFilter"}},
    "formatters": {
        "json": {"()": "observability.errors.StructuredExceptionFormatter"},
    },
    "handlers": {
        "stdout": {"class": "logging.StreamHandler", "formatter": "json"},
    },
    "root": {"level": "INFO", "handlers": ["stdout"], "filters": ["redact"]},
}

Driving it with a chained failure produces one record that a backend can group by outer type and drill into by inner type:

logging.config.dictConfig(CONFIG)
install_hooks(logging.getLogger("service"))

try:
    try:
        {"a": 1}["user_id"]
    except KeyError as exc:
        raise ValueError("malformed payload; token=abc123") from exc
except ValueError:
    logging.getLogger("service").exception("request failed")

Expected Output:

{
  "ts": "2026-08-02T11:04:19+0000",
  "level": "ERROR",
  "logger": "service",
  "message": "request failed",
  "exception": {
    "type": "ValueError",
    "module": "builtins",
    "message": "malformed payload; token=***",
    "frames": [
      {"file": "app.py", "line": 42, "func": "<module>", "code": "raise ValueError(...) from exc"}
    ],
    "cause": {
      "type": "KeyError",
      "module": "builtins",
      "message": "'user_id'",
      "frames": [
        {"file": "app.py", "line": 40, "func": "<module>", "code": "{\"a\": 1}[\"user_id\"]"}
      ]
    }
  }
}

The token is masked, the chain survives as a nested object rather than a wall of text, and exception.type is a field you can group an alert on.

Common mistakes

The traceback arrives as one opaque string

Error signature: the log backend shows message containing the whole traceback, and faceting by exception type is impossible. Root cause: the default Formatter appends formatException() output to the formatted message; nothing splits it into fields. Remediation: override formatException() to return serialised fields, and emit them under their own key as in Step 2. Group alerts on exception.type, not on message text.

A second handler prints the first handler's format

Error signature: the plain-text file sink contains JSON tracebacks, or vice versa. Root cause: Formatter.format() caches its render on record.exc_text and every later formatter reuses it — records are shared across handlers. Remediation: set record.exc_text = None at the top of each formatter that renders exceptions itself. If two handlers genuinely need different renderings, that reset is mandatory, not optional.

exc_info is None on the far side of a queue

Error signature: the listener's formatter emits "exception": null while the message field contains a traceback. Root cause: QueueHandler.prepare() formats, copies, and strips the record so it can be pickled. Remediation: for an in-process queue.Queue, override prepare() to return the record unchanged. Across processes, accept the strip and put the formatter you want on the producing side.

Logging the same exception at four levels of the stack

Error signature: one incident produces four ERROR records with four different messages and four partial tracebacks. Root cause: every layer catches, logs, and re-raises. Remediation: catch to translate — raise NewError from exc — and log once, at the boundary that decides the outcome. The chain preserves everything the intermediate logs were trying to say.

A traceback keeps a request alive in memory

Error signature: memory climbs after a burst of errors and does not come back down; heap dumps show request payloads pinned by frame objects. Root cause: a traceback references every frame, each frame references its locals, and the code stashed exc_info or the exception object in a module-level list for later reporting. Remediation: format and release inside the except block. If you must keep the exception, strip it first with exc.with_traceback(None) and keep only the serialised form.

Related

• Python logging fundamentals and structured data — the parent section: logger hierarchy, record schema, and the full handler graph.
• Logging exceptions and tracebacks in Python — the concrete walkthrough of exc_info, chains, and frame limits.
• Capturing unhandled exceptions and warnings — the four hooks that catch what never reaches an except block.
• Redacting sensitive data in log records — masking secrets in messages, exception args, and structured fields.
• Handler architecture for Python logging — where filters and formatters sit relative to the queue boundary.
• Adding trace IDs to log records — the correlation fields that make an error record clickable through to its trace.

Frequently Asked Questions