Logging Exceptions and Tracebacks in Python

You need the traceback in your log, not a message that says an error occurred — and you need it as fields your backend can group on rather than a multi-line blob. This walkthrough is for engineers wiring up a service's error path for the first time, or fixing one that lost its stack somewhere between the except block and the sink. It builds on the exception and traceback logging guide and is part of the Python logging fundamentals and structured data section.

Everything here turns on one fact: exc_info on a log record is a live (type, value, traceback) triple, not text. What you do with that triple before it reaches a formatter decides whether an incident is one queryable record or a wall of prose.

Prerequisites

The standard library covers everything on this page. Pin a JSON serialiser only if you want the structured output shown below in a service that does not already have one.

pip install "python-json-logger>=2.0.7,<4.0.0"

export LOG_TRACEBACK_FRAMES=20     # read by the formatter below

Implementation

Step 1 — Attach the exception at the boundary that handles it. logger.exception() is logger.error() with exc_info=True; inside an except block it picks up the active exception with its traceback and its full chain. Outside one, pass the object you hold.

import logging

logger = logging.getLogger(__name__)

def load_user(row: dict) -> dict:
    try:
        return {"id": row["user_id"], "email": row["email"]}
    except KeyError as exc:
        raise ValueError("row is missing a required column") from exc   # keep the link

def handler(rows: list[dict]) -> None:
    for row in rows:
        try:
            load_user(row)
        except ValueError:
            logger.exception("row rejected", extra={"row_index": rows.index(row)})

One log call, at the layer that decides to skip the row. load_user translates instead of logging, so the record carries ValueError with the KeyError attached underneath as __cause__.

Step 2 — Slice the frame list from the innermost end. traceback.extract_tb() returns frames outermost-first, so a naive [:20] keeps twenty frames of WSGI, middleware and routing and drops the line that actually failed. Slice with [-20:] instead.

import traceback

FRAMES = 20

def frames_of(exc: BaseException) -> list[dict]:
    """Innermost FRAMES stack entries — the ones nearest the failure."""
    entries = traceback.extract_tb(exc.__traceback__)
    kept = entries[-FRAMES:]
    return [
        {"file": f.filename, "line": f.lineno, "func": f.name, "code": f.line}
        for f in kept
    ]

Step 3 — Walk the chain. __cause__ is set by raise ... from exc; __context__ is set implicitly when an exception is raised inside an except block; __suppress_context__ is set by raise ... from None. Prefer cause, fall back to context, and stop at a fixed depth so a pathological chain cannot run away.

def serialise(exc: BaseException, depth: int = 0) -> dict:
    node = {
        "type": type(exc).__name__,
        "module": type(exc).__module__,
        "message": str(exc)[:512],
        "frames": frames_of(exc),
    }
    if depth >= 3:
        node["chain_truncated"] = True
        return node
    if exc.__cause__ is not None:
        node["cause"] = serialise(exc.__cause__, depth + 1)
    elif exc.__context__ is not None and not exc.__suppress_context__:
        node["context"] = serialise(exc.__context__, depth + 1)
    return node

Keeping cause and context under different keys is what lets you alert on the second one. A context link means something failed while handling a failure — a rollback, a cleanup, a finally — and that is a distinct class of bug worth paging on separately.

Step 4 — Render fields instead of prose. Override formatException() and clear the cached exc_text first, or this formatter may be handed a rendering some other handler produced from the same shared record.

import json
import logging

class ExceptionFieldsFormatter(logging.Formatter):
    def formatException(self, ei) -> str:
        return json.dumps(serialise(ei[1]), default=str)

    def format(self, record: logging.LogRecord) -> str:
        record.exc_text = None                       # do not inherit another handler's render
        out = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        for key, value in getattr(record, "__dict__", {}).items():
            if key == "row_index":                   # promote your own extras explicitly
                out[key] = value
        if record.exc_info:
            out["exception"] = json.loads(self.formatException(record.exc_info))
        return json.dumps(out, default=str)

Configuration options

Verification

Run a chained failure through the formatter and check three things: the exception is its own object rather than text in the message, the chain survived, and the innermost frame is the line that raised.

import logging, sys

h = logging.StreamHandler(sys.stdout)
h.setFormatter(ExceptionFieldsFormatter())
log = logging.getLogger("verify")
log.addHandler(h)
log.setLevel(logging.INFO)

try:
    try:
        {"a": 1}["user_id"]
    except KeyError as exc:
        raise ValueError("row is missing a required column") from exc
except ValueError:
    log.exception("row rejected", extra={"row_index": 7})

Expected Output:

{"ts": "2026-08-02T11:22:03+0000", "level": "ERROR", "logger": "verify",
 "message": "row rejected", "row_index": 7,
 "exception": {"type": "ValueError", "module": "builtins",
   "message": "row is missing a required column",
   "frames": [{"file": "verify.py", "line": 16, "func": "<module>",
               "code": "raise ValueError(\"row is missing a required column\") from exc"}],
   "cause": {"type": "KeyError", "module": "builtins", "message": "'user_id'",
     "frames": [{"file": "verify.py", "line": 14, "func": "<module>",
                 "code": "{\"a\": 1}[\"user_id\"]"}]}}}

Lock it in with a test so a later refactor cannot quietly drop the chain:

def test_chain_survives(caplog):
    with caplog.at_level(logging.ERROR):
        try:
            try:
                raise KeyError("user_id")
            except KeyError as exc:
                raise ValueError("bad row") from exc
        except ValueError:
            logging.getLogger("t").exception("failed")
    record = caplog.records[0]
    data = serialise(record.exc_info[1])
    assert data["type"] == "ValueError"
    assert data["cause"]["type"] == "KeyError"      # the link is the point

Expected Output:

test_exceptions.py::test_chain_survives PASSED

Common mistakes

NoneType: None where the traceback should be

Error signature: the record ends with the literal text NoneType: None. Root cause: logger.exception() or exc_info=True was called outside an except block, so sys.exc_info() returned an empty triple. Remediation: call it only from an except block; elsewhere pass the object — logger.error("failed", exc_info=exc). Logging from a callback or a Future result almost always needs the second form.

The traceback stops at the re-raise

Error signature: one frame, pointing at your own raise statement, with nothing underneath. Root cause: the code caught an exception and raised a fresh one with a bare raise NewError(...) at the top of the stack, so the new object's traceback begins there. Remediation: write raise NewError(...) from exc. The chain link carries the original traceback, which is where the deeper frames live.

Every truncated traceback looks identical

Error signature: hundreds of distinct failures share the same twenty frames of middleware and none names an application file. Root cause: the frame list was sliced from the front, keeping the outermost frames. Remediation: slice with [-N:] and record frames_total alongside a truncated flag so the shortening is visible in the record.

Operating it

The structured form pays off in three specific ways once it is running, and each one is worth setting up deliberately rather than hoping the backend does it for you.

Grouping. With exception.type and the innermost frame's file and line as separate fields, a backend can group by the tuple and give you one row per distinct failure rather than one row per occurrence. That single change turns "eight thousand errors overnight" into "three distinct failures, one of which accounts for 97% of the volume", which is the first question anyone asks. Group on the type plus the innermost frame rather than on the message: the message frequently contains an identifier, so grouping by it produces one group per request and hides the pattern completely.

Alerting on the chain shape. A context link — an exception raised while another was being handled — is a distinct class of bug, and it is worth its own alert precisely because it is rare. A rollback that fails during error handling, a finally block that raises, a cleanup that assumes a resource which was never acquired: all of them surface as a context link and none of them surface as anything else. A query for records where exception.context exists, alerting on any occurrence, is one of the cheapest high-signal alerts available.

Retention. Serialised exceptions are large compared with ordinary records — a twenty-frame traceback with source lines is several kilobytes — so an error burst can cost more storage than a day of normal traffic. The frame limit, the chain depth and the message truncation from the implementation section exist for that reason, and they are worth revisiting once you can see real volume. A useful shape is to keep the full serialised exception for ERROR and above and to strip frames from anything lower, since a DEBUG record with a traceback is almost always incidental.

Custom exception types are part of the instrumentation

One design decision does more for grouping than any formatter change: raise domain-specific exception types rather than reusing ValueError everywhere. OrderRejected, PaymentDeclined and InventoryUnavailable group cleanly, alert independently and read correctly in a dashboard; three ValueErrors with different messages do none of that, and no amount of serialisation fixes it. The exception type is the field the whole error pipeline is keyed on, so it is worth choosing rather than defaulting to whatever the standard library offers.

The corollary is to keep the hierarchy shallow. A base AppError with a handful of direct subclasses gives you both a specific type for grouping and a general one for except clauses. A five-level hierarchy gives you neither, because every alert has to decide which level to key on and different people choose differently.

Related

• Exception and traceback logging in Python — the parent guide: exc_text caching, chain semantics, and the queue boundary.
• Capturing unhandled exceptions and warnings — the hooks that catch what never reaches an except block.
• Redacting sensitive data in log records — masking secrets that ride along in exception messages.
• Structured logging with the Python standard library — the JSON formatter this page extends.
• Recording exceptions and span events — the same failure, attached to the trace instead of the log.

Frequently Asked Questions