Handling Multi-Line Tracebacks in Log Shippers

A Python traceback is one event that prints as a dozen lines, and every line-oriented shipper in existence will happily index it as a dozen unrelated documents. The result is a store in which searching for TimeoutError returns a document containing only the words TimeoutError: read timed out, with no service, no request, and no indication of which line raised it. This page covers the two fixes, why one of them is permanent, and how to verify either. It is a task article under log shipping and collection, part of the Python telemetry pipelines and delivery section, and it builds on logging exceptions and tracebacks in Python.

Prerequisites

pip install "python-json-logger>=2.0.7,<4.0.0" \
            "structlog>=24.1.0,<26.0.0"

Implementation

Step 1 — Format the exception into a field. The standard library already produces the text; the only change is putting it somewhere structured instead of letting it reach the stream as extra lines. Every JSON formatter has a hook for this, and once it is in place the entire class of problem disappears for every record the application produces.

import logging
from pythonjsonlogger import jsonlogger


class ExceptionAsFieldFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, target, record, message_dict):
        super().add_fields(target, record, message_dict)
        target["level"] = record.levelname
        target["logger"] = record.name
        if record.exc_info:
            # 1. The whole traceback becomes one string value.
            target["exception"] = self.formatException(record.exc_info)
            target["exception_type"] = record.exc_info[0].__name__
            target.pop("exc_info", None)
        if record.stack_info:
            target["stack"] = self.formatStack(record.stack_info)

Step 2 — Escape newlines on the way out, unconditionally. Step 1 puts the traceback in a field; JSON serialisation escapes the newlines inside it. But a message argument, a third-party string or a user-supplied value can still carry a raw newline, and one of those undoes everything. Escaping at the formatter's exit is two lines and removes the possibility entirely.

    def format(self, record: logging.LogRecord) -> str:
        line = super().format(record)
        return line.replace("\n", "\\n").replace("\r", "")

Expected Output: a chained exception, complete, on one line.

{"asctime": "2026-09-18T09:41:07.512Z", "level": "ERROR", "logger": "billing", "message": "charge failed", "exception_type": "TimeoutError", "exception": "Traceback (most recent call last):\n  File \"billing.py\", line 88, in charge\n    gateway.post(payload)\n  File \"http.py\", line 41, in post\n    raise TimeoutError(\"read timed out\")\nTimeoutError: read timed out\n\nThe above exception was the direct cause of:\n\nTraceback (most recent call last):\n  File \"api.py\", line 22, in handler\n    charge(order)\nChargeFailed: gateway unavailable"}

Step 3 — Add a shipper rule only for output you do not control. The interpreter writes uncaught tracebacks directly to stderr, and some libraries print rather than log. Those lines never pass through your formatter, so the shipper is the only place they can be reassembled. The rule matches the shape of a first line and treats everything that does not match as continuation of the previous record.

[MULTILINE_PARSER]
    Name          python_traceback
    Type          regex
    Flush_Timeout 1000
    # A record starts with a timestamp; anything else continues the one before it.
    Rule  "start_state"  "/^\d{4}-\d{2}-\d{2}[ T]/"       "cont"
    Rule  "cont"         "/^(?!\d{4}-\d{2}-\d{2}[ T]).*/" "cont"

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Multiline.Parser python_traceback

Step 4 — Anchor on the record start, not on indentation. The common mistake is a rule that treats indented lines as continuation. It works on the frame lines and fails on the last line of a traceback, which is not indented, so the exception type ends up in a document of its own — the single most important line, separated from its context. Matching "does this look like the start of a new record" is the formulation that survives contact with real output.

Step 5 — Bound the reassembly. Every multi-line parser needs a flush timeout and a size ceiling. Without them, a run of unmatched lines accumulates into one enormous record, and a shipper holding an unbounded buffer waiting for a start line that never comes is a memory leak with a very confusing symptom.

Why the formatter fix is permanent and the parser fix is not

Both approaches produce one document per failure, so it is reasonable to ask why the choice matters. It matters because the two fixes live in different places and decay differently.

The formatter fix lives in the application, applies to every record that application produces, and travels with it. A service moved from one cluster to another, from one shipper to another, or from a file-based pipeline to an OTLP one keeps working, because the guarantee — one record is one line — is made at the point of production and never depends on anything downstream being configured correctly.

The parser fix lives in the collector, and there is one collector configuration per environment. It has to be present in production, in staging, in the developer's local compose file and in whatever the next platform migration introduces. It has to survive shipper upgrades, since multi-line syntax has changed more than once. And it is silently version-dependent on the output it parses: a library that changes its log prefix, or a Python release that changes traceback formatting — as the addition of caret markers under the failing expression did — can break a rule that has worked for years, with no error anywhere, producing exactly the fragmentation it was written to prevent.

The practical conclusion is not that parsers are wrong but that their scope should be small. Fix the application's own records in the formatter, where the fix is permanent, and reserve the parser for the residue: interpreter output on stderr, third-party libraries that print, and anything else you cannot reach. That residue is small enough that when its rule does break, the damage is bounded to output nobody was querying by field anyway. The same logic is why exporting Python logs through OpenTelemetry removes the problem rather than solving it: there are no physical lines in a protocol message.

One further reason to shrink the residue rather than perfect the rule: multi-line reassembly is stateful, and state in a collector is where ordering guarantees go. A reassembling input holds a partial record while it waits for the next line, which means records from that container are emitted out of order relative to records from others, and a restart mid-reassembly discards whatever was being accumulated. Neither effect is large, and neither is worth reasoning about if the application's own records never need reassembly in the first place. That is the whole argument in one sentence: a stateless path cannot lose state.

Configuration options

Verification

Test with a chained exception, because the simple case passes under rules that break on the real one.

import logging

def verify():
    try:
        try:
            raise TimeoutError("read timed out")
        except TimeoutError as exc:
            raise RuntimeError("gateway unavailable") from exc
    except RuntimeError:
        logging.getLogger("verify").exception("charge failed", extra={"order": "ord_1"})

verify()

Expected Output: exactly one physical line, containing both exception types.

python verify.py 2>&1 | wc -l          # must print 1
python verify.py 2>&1 | python3 -c "import json,sys; print(json.load(sys.stdin)['exception_type'])"

1
RuntimeError

If wc -l prints more than one, step 2 is not in effect. If the JSON parse fails, something in the record carried a raw newline that the escape did not reach — usually a field added after the formatter ran.

Common mistakes

A rule that works locally fragments in production. Error signature: single records in development, split records in the cluster. Root cause: the runtime envelope prefixes each line, so the rule's anchor never matches. Remediation: apply the runtime parser before the multi-line parser, so the rule sees the application's own line.

The exception type lands in its own document. Error signature: searching for an exception class returns context-free records. Root cause: an indentation-based continuation rule. Remediation: anchor on the start of a record instead, as in step 4.

One record grows to megabytes. Error signature: a store rejecting an oversized document, or a shipper's memory climbing steadily. Root cause: a start-line pattern that stopped matching, so every subsequent line is treated as continuation. Remediation: set a flush timeout and a buffer ceiling, and alert on records above a size threshold.

Tracebacks are intact but unsearchable. Error signature: the field exists and a query for an exception type matches nothing. Root cause: the whole traceback is one analysed text blob, or the exception type is only inside it. Remediation: emit exception_type as a separate keyword field, as step 1 does, so the common query does not depend on full-text matching.

Frequently Asked Questions