Renaming and Dropping Fields in structlog

structlog builds each event as a dictionary and hands it through a chain of processors before rendering, which makes it well suited to shaping output to an exact schema. Its defaults do not match most schemas — the message is under event, application fields sit at the top level alongside everything else — and matching them is a single processor, placed correctly. This page covers renaming structlog's keys, namespacing application fields, dropping what must never leave the process, and the placement that stops anything slipping past. It is a task article under structlog processors and pipelines, part of the modern Python logging libraries deep dive section, and it applies the conventions from JSON log schemas and conventions.

Prerequisites

pip install "structlog>=24.1.0,<26.0.0"

Implementation

Step 1 — Rename structlog's own keys. The keys structlog and its standard processors produce — event, level, timestamp, logger — have fixed names that usually differ from the schema's. A mapping from structlog's names to the schema's, applied in a processor, translates them without changing any call site.

RENAME = {
    "event": "message",
    "level": "log.level",
    "timestamp": "@timestamp",
    "logger": "log.logger",
    "exception": "error.stack_trace",
}

Step 2 — Namespace application fields. Every key that is not a shared schema field goes under one object. This is the rule that prevents a service's status from colliding with a shared status, and it requires a list of the shared names so the processor knows which keys stay at the top level.

SHARED = {
    "message", "log.level", "@timestamp", "log.logger", "error.stack_trace",
    "service.name", "service.version", "deployment.environment",
    "trace_id", "span_id", "request_id",
    "http.request.method", "http.route", "http.response.status_code", "duration_ms",
}

Step 3 — Drop or redact what must never leave. Key names that indicate secrets are redacted wherever they appear, including inside the namespace. Doing this in the same processor that shapes the output means there is exactly one place it happens and nothing can be added after it.

SENSITIVE = {"password", "passwd", "secret", "token", "authorization", "api_key", "card_number"}
REDACTED = "[REDACTED]"


def shape_for_schema(logger, method_name, event_dict: dict) -> dict:
    out: dict = {}
    app: dict = {}
    for key, value in event_dict.items():
        name = RENAME.get(key, key)
        if key.lower() in SENSITIVE:
            value = REDACTED                                  # 1. redacted wherever it lands
        if name in SHARED:
            out[name] = value                                 # 2. shared field, schema name
        elif key == "app" and isinstance(value, dict):
            app.update(value)
        else:
            app[key] = value                                  # 3. everything else, namespaced
    if app:
        out["app"] = app
    return out

Step 4 — Place it immediately before the renderer. Processors run in order, and a processor that runs after the shaping step adds keys under their own names, outside the namespace and unredacted. The shaping processor therefore belongs as late as possible — after timestamping, after exception formatting, after any processor that adds context — and immediately before rendering.

import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.format_exc_info,
        add_trace_context,                    # adds trace_id / span_id
        shape_for_schema,                     # 4. last before rendering
        structlog.processors.JSONRenderer(),
    ],
    cache_logger_on_first_use=True,
)

Expected Output: a record in the schema's shape, with the secret gone.

{"message": "order accepted", "log.level": "info", "@timestamp": "2026-09-18T14:02:11.408Z", "log.logger": "orders", "request_id": "9c1f4e7a", "trace_id": "9f2a71c4f0b84c2e9d5f1a7b3c8e6d02", "app": {"order_id": "ord_7", "item_count": 3, "password": "[REDACTED]"}}

Step 5 — Test the processor directly. A processor is a plain function taking a dictionary and returning one, which makes it trivially testable without any logging configuration. Tests that feed it representative event dictionaries — including ones with sensitive keys and ones with keys that collide with shared names — verify the mapping precisely and run in microseconds.

def test_shape_renames_namespaces_and_redacts():
    out = shape_for_schema(None, "info", {
        "event": "x", "level": "info", "status": "held", "token": "abc123"})
    assert out["message"] == "x" and out["log.level"] == "info"
    assert "status" not in out and out["app"]["status"] == "held"
    assert out["app"]["token"] == "[REDACTED]"

When structlog hands off to the standard library

Many services configure structlog to pass its events to the standard library for output, so that structlog events and records from other libraries share one handler. In that arrangement the shaping can happen in two places, and choosing one matters.

In the structlog chain, before handoff. The processor runs only on structlog events. Records from libraries logging through the standard library bypass it and arrive in their own shape. That is acceptable when the handler's formatter handles those records separately, and it means two places define the output shape — one for structlog events, one for everything else.

In the formatter, after handoff. structlog's ProcessorFormatter accepts a list of processors that run on every record the handler formats — both structlog events and foreign records converted into event dictionaries. Placing the shaping processor there, immediately before the renderer, applies one mapping to all output from the process. This is usually the better arrangement, because it gives the whole service a single definition of its output shape, including records from third-party libraries.

formatter = structlog.stdlib.ProcessorFormatter(
    foreign_pre_chain=[
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
    ],
    processors=[
        structlog.stdlib.ProcessorFormatter.remove_processors_meta,
        shape_for_schema,                      # one shape for every record
        structlog.processors.JSONRenderer(),
    ],
)

Whichever is chosen, it should be exactly one. Shaping in both places applies the rename twice — harmless for most keys and confusing for any that are renamed into a name another rule also matches.

Dropping versus redacting

The processor above redacts sensitive keys — it keeps the key and replaces the value. Dropping them entirely is the other option, and the choice has consequences worth thinking through.

Redaction preserves the fact that the field was present. An operator reading a record with token: [REDACTED] knows a token was involved in this code path, which is sometimes diagnostic in itself — a request that should not have carried a token did, or one that should have did not. The redacted value also makes it obvious when redaction is working, which is useful in reviews and tests.

Dropping removes all trace of the field. It is appropriate for fields that should never have been logged in the first place and whose presence carries no information — a whole request body passed by mistake, for example, where even a placeholder would be misleading about what the record contains. It also shrinks the record, which matters for large accidental fields.

A common arrangement is to redact known secret names and drop known bulk fields: tokens and passwords become placeholders, while keys like body, payload or raw are removed with a counter incremented so their removal is still visible in aggregate. Whichever is chosen, it should be applied consistently in the one processor, so the output never depends on which call site produced the record.

Evolving the mapping safely

The mapping tables in this processor are, in effect, the service's implementation of the fleet schema, and they change over time. Three practices keep those changes safe.

Keep the tables in the shared logging package, not in each service. A mapping duplicated across services drifts, and the drift is exactly the inconsistency the schema was meant to remove. A single definition imported everywhere changes everywhere at once.

Treat an addition to SHARED as a schema change. Moving a key from the namespace to the top level changes where every consumer finds it. That needs the same notice as any other rename: emit both locations for a transition period, then remove the old one, as described in designing a log schema for a service fleet.

Extend SENSITIVE freely. Adding a name to the redaction set has no compatibility cost and closes a leak. It should be the easiest change in the whole logging configuration to make, and a new sensitive field discovered in production should lead to an addition here the same day.

Configuration options

Verification

Render a representative event through the configured pipeline and check the shape.

import json, io, structlog

buf = io.StringIO()
structlog.configure(processors=[*PROCESSORS], logger_factory=structlog.PrintLoggerFactory(buf))
structlog.get_logger("orders").info("order accepted", order_id="ord_7", token="abc")
rec = json.loads(buf.getvalue())
print(sorted(rec), rec["app"]["token"])

Expected Output:

['@timestamp', 'app', 'log.level', 'log.logger', 'message'] [REDACTED]

Common mistakes

Shaping before other processors. Error signature: some keys renamed and others not. Root cause: processors after the shaping step add keys it never saw. Remediation: place it immediately before the renderer.

No namespace for application keys. Error signature: a service field overwriting a shared one of the same name. Root cause: all keys at the top level. Remediation: move non-shared keys under one object.

Redaction in one place, shaping in another. Error signature: a sensitive key surviving because it was added between the two. Root cause: two processors with different positions. Remediation: redact in the shaping processor.

Shaping only structlog events. Error signature: third-party library records in a different shape. Root cause: the processor runs before handoff to the standard library. Remediation: put it in the ProcessorFormatter's processors.

Shaping twice. Error signature: keys renamed into unexpected names. Root cause: the same mapping applied both before and after handoff. Remediation: exactly one location.

Frequently Asked Questions