Migrating from Standard Logging to structlog: Zero-Downtime Config and Diagnostics

You need to move a running service from standard library logging calls to structured JSON output from structlog without double-emitting records, losing context, or taking downtime during the cutover. This guide gives the exact bridging configuration, processor ordering, and validation harness. It builds on Structlog Architecture and Setup and is part of the Modern Python Logging Libraries Deep Dive.

Prerequisites

Pin structlog and an OpenTelemetry API for the trace-context step.

pip install \
  "structlog>=24.1.0,<26.0.0" \
  "opentelemetry-api>=1.30.0,<2.0.0"

Run configuration at import time, before any logger is fetched, so cached loggers pick up the chain. Set the service log level through the environment if you gate output by severity.

export LOG_LEVEL="INFO"

Implementation

The migration is zero-downtime precisely because the stdlib factory bridges rather than replaces: existing logging.getLogger(__name__).info(...) calls keep working and start emitting JSON the moment configuration runs, so you can cut over a running service by deploying a configuration change rather than rewriting call sites. The risk is not the call sites; it is the handler topology. Standard logging propagates records up to the root logger, and any library that logs — your web framework, the database driver, an HTTP client — feeds the same root. If both a leftover root handler and the structlog pipeline are attached, every one of those records emits twice. The first step therefore removes existing handlers before anything else.

Step 1 — Clear root handlers and install the stdlib factory. Standard logging attaches a default StreamHandler to the root logger. Adding a structlog handler on top double-emits, so remove the existing handlers first, then configure structlog with structlog.stdlib.LoggerFactory and BoundLogger. This routes legacy logging.getLogger calls through the same processor chain without touching call sites.

import logging
import structlog

# Remove the default root handler to prevent double emission
for handler in logging.root.handlers[:]:
    logging.root.removeHandler(handler)

structlog.configure(
    wrapper_class=structlog.stdlib.BoundLogger,
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    cache_logger_on_first_use=True,
)

structlog.get_logger().info("service_started", version="2.1.0")

Expected Output:

{"event": "service_started", "level": "info", "logger": "__main__", "timestamp": "2024-01-15T10:00:00.000000Z", "version": "2.1.0"}

Step 2 — Order the processor chain and inject trace context. Processor order dictates serialization: merge async context first, add level and timestamp, format exceptions and stack info before the renderer, then render JSON last. Inject trace_id and span_id via bind_contextvars so they survive await boundaries.

import structlog
from opentelemetry import trace

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.JSONRenderer(),
    ],
)


def inject_otel_context():
    ctx = trace.get_current_span().get_span_context()
    if ctx.is_valid:
        structlog.contextvars.bind_contextvars(
            trace_id=f"{ctx.trace_id:032x}",
            span_id=f"{ctx.span_id:016x}",
            trace_flags=int(ctx.trace_flags),
        )


inject_otel_context()
structlog.get_logger().info("request_processed", endpoint="/api/v1/data")

Expected Output:

{"event": "request_processed", "level": "info", "timestamp": "2024-01-15T10:00:01.000000Z", "trace_id": "00000000000000000000000000000001", "span_id": "0000000000000002", "trace_flags": 1, "endpoint": "/api/v1/data"}

Step 3 — Handle legacy formatting and exception traces. Legacy code passes positional %s arguments; structlog expects keyword arguments for clean JSON keys, and BoundLogger auto-converts the positional string. Multi-line tracebacks need structlog.processors.format_exc_info in the chain; exc_info=True alone does not serialize frames into the exception key. Validate the schema so a stack_info key does not collide with exception.

For a phased rollout across many services, run the new pipeline behind a feature flag and direct the structlog JSON stream to stdout while leaving any legacy file handler in place but isolated, so the two never share a destination. Once you have confirmed the JSON parses cleanly in the aggregator and the trace fields correlate to spans, remove the legacy handler. The contextvars-based trace injection in step two is what makes the migrated logs useful rather than merely structured: a record without trace_id is structured noise, while a record carrying the active span's identifiers joins directly to the trace waterfall in your backend. Bind those identifiers once at request entry — in middleware or an ASGI lifespan dependency — and merge_contextvars carries them into every record the request produces, including records emitted by bridged stdlib loggers deep in third-party code.

Incremental migration with ProcessorFormatter

The configuration in step one routes records through structlog's own LoggerFactory, which is the right end state but a large blast radius for a first deploy. The lower-risk path keeps the standard library's handler topology exactly as it is and changes only the formatter. A structlog.stdlib.ProcessorFormatter can render any LogRecord — including ones that never touched structlog — through a shared processor chain, so you flip the output to JSON before you change a single call site, then adopt structlog.get_logger module by module on your own schedule.

The key is one shared processor chain used in two places: as the foreign_pre_chain for foreign stdlib records, and as the prefix of the native structlog chain. Define it once so the two paths can never drift.

import logging
import structlog

# One chain, reused for both native structlog events and foreign stdlib records.
shared_processors = [
    structlog.contextvars.merge_contextvars,
    structlog.stdlib.add_logger_name,
    structlog.stdlib.add_log_level,
    structlog.processors.TimeStamper(fmt="iso"),
    structlog.processors.StackInfoRenderer(),
    structlog.processors.format_exc_info,
]

# Native structlog ends by handing the event to the stdlib formatter.
structlog.configure(
    processors=shared_processors + [
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

# One ProcessorFormatter renders BOTH sources to flat JSON.
formatter = structlog.stdlib.ProcessorFormatter(
    foreign_pre_chain=shared_processors,   # replays the chain on non-structlog records
    processors=[
        structlog.stdlib.ProcessorFormatter.remove_processors_meta,
        structlog.processors.JSONRenderer(),
    ],
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)
root = logging.getLogger()
root.handlers[:] = [handler]   # replace, do not append, to avoid double emission
root.setLevel(logging.INFO)

# Foreign caller that has NOT been migrated yet — still emits JSON.
logging.getLogger("legacy.payments").warning("charge_retry", extra={"attempt": 2})
# Already-migrated caller.
structlog.get_logger("orders").info("order_created", order_id="ORD-9")

Expected Output:

{"attempt": 2, "logger": "legacy.payments", "level": "warning", "timestamp": "2024-01-15T10:00:02.000000Z", "event": "charge_retry"}
{"logger": "orders", "level": "info", "timestamp": "2024-01-15T10:00:02.001000Z", "order_id": "ORD-9", "event": "order_created"}

Both lines share the same keys, the same timestamp format, and the same level vocabulary because they ran the same shared_processors. That is what keeps third-party libraries consistent: a SQL warning from your ORM, a connection-pool message from an HTTP client, and your own migrated events all reach the aggregator as one schema, so a single parser and a single set of dashboard fields cover everything. You never have to migrate the library; its LogRecord simply flows through the foreign_pre_chain. The rollout then becomes safe and monotonic — ship the formatter swap first and watch the JSON validate, then convert call sites whenever a module is already being touched, with no flag-day cutover and no window where half the stream is plain text and half is JSON.

Configuration options

When you are still weighing structlog against the alternatives before committing to this migration, compare it with Loguru and the standard library in structlog vs Loguru vs standard library logging, and for the trace-correlated microservice angle see Loguru vs structlog for microservices.

Verification

Use structlog.testing.LogCapture in CI to assert the structured record without serializing to a real sink. This catches schema regressions before deploy.

import structlog.testing

capture = structlog.testing.LogCapture()
structlog.configure(
    processors=[capture],
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=False,
)

structlog.get_logger().info("test_migration", status="pass")

assert len(capture.entries) == 1
assert capture.entries[0]["status"] == "pass"
print("Validation successful.")

Expected Output:

Validation successful.

Common mistakes

• Double-serialization of records. Symptom: {"message": "{\"event\": \"test\", \"level\": \"info\"}"}. Cause: both a logging.Formatter and the structlog JSONRenderer serialize the same record. Remediation: remove all formatters from handlers and let structlog own serialization.
• Late structlog.configure(). Symptom: TypeError: 'Logger' object is not callable or missing processors in output. Cause: configuring after the first get_logger() leaves cached loggers on the old chain. Remediation: configure at import time, before any logging call.
• Ignoring exc_info in async contexts. Symptom: missing exception key in JSON. Cause: a format_exc_info processor is absent, or the log call runs outside the except block. Remediation: add format_exc_info to the chain and call the logger with exc_info=True from inside the handler.
• Foreign records drift from native events during incremental migration. Symptom: third-party LogRecord lines lack timestamp or carry a different level field than your migrated calls. Cause: the foreign_pre_chain and the native structlog processors were defined as two separate lists that fell out of sync. Remediation: define one shared_processors list and reference it in both structlog.configure and the ProcessorFormatter(foreign_pre_chain=...), so the two code paths cannot diverge.

Frequently Asked Questions