Redacting Sensitive Data in Python Log Records

A connection string in an exception message, a bearer token interpolated into a debug line, a customer record in a repr() — every one of those is a log entry you cannot delete from a retention system after the fact. This page shows the filter that stops them, where it has to sit to be effective, and the three fields most implementations forget. It builds on the exception and traceback logging guide and is part of the Python logging fundamentals and structured data section.

Placement matters more than the pattern. A perfect regex on the wrong side of a queue protects nothing.

Prerequisites

The standard library is enough. Pin a structured formatter if the service does not already emit JSON, since redaction and structure are usually adopted together.

pip install "python-json-logger>=2.0.7,<4.0.0"

export LOG_REDACT_SALT="$(openssl rand -hex 16)"   # per-deployment, not per-process

The salt must be identical across every replica of a service and rotate with your normal secret rotation. A per-process salt makes hashes uncorrelatable, which defeats the reason for hashing at all.

Implementation

Step 1 — Redact by key first, by value second. Key-based redaction — "the field called password is always masked" — has no false positives and no false negatives for the fields you name. Value-based patterns are for formats that cannot be anything else: a Bearer header, an sk- prefixed key, a PEM block, a card number that passes a Luhn check. Anything looser will eventually mask a request ID and hide the field you needed.

import hashlib
import logging
import os
import re

SECRET_KEYS = frozenset({
    "password", "passwd", "secret", "token", "api_key", "apikey",
    "authorization", "cookie", "session", "private_key", "card_number",
})

VALUE_PATTERNS = (
    re.compile(r"\bBearer\s+[A-Za-z0-9._\-]+", re.I),
    re.compile(r"\bsk-[A-Za-z0-9]{16,}\b"),
    re.compile(r"://[^:/@\s]+:[^@\s]+@"),                  # credentials in a DSN
    re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]+?-----END [A-Z ]*PRIVATE KEY-----"),
)

_SALT = os.environ.get("LOG_REDACT_SALT", "").encode()

def fingerprint(value: str) -> str:
    """Stable across replicas, useless without the salt."""
    return "sha256:" + hashlib.sha256(_SALT + value.encode()).hexdigest()[:12]

Step 2 — Cover all four fields that carry text. This is where most implementations fall short. A log call carries its data in more than one place, and a filter that rewrites only record.msg leaves three doors open.

class RedactFilter(logging.Filter):
    """Runs on the producing thread; mutates the record in place and returns True."""

    def filter(self, record: logging.LogRecord) -> bool:
        record.msg = self._scrub(record.msg)                       # 1. the format string
        if record.args:                                            # 2. the args tuple
            if isinstance(record.args, dict):
                record.args = {k: self._by_key(k, v) for k, v in record.args.items()}
            else:
                record.args = tuple(self._scrub(a) for a in record.args)
        for key in list(vars(record)):                             # 3. structured extras
            if key in SECRET_KEYS:
                setattr(record, key, fingerprint(str(getattr(record, key))))
            elif key not in logging.LogRecord("", 0, "", 0, "", (), None).__dict__:
                setattr(record, key, self._scrub(getattr(record, key)))
        if record.exc_info and record.exc_info[1] is not None:     # 4. exception args
            exc = record.exc_info[1]
            if exc.args and isinstance(exc.args[0], str):
                exc.args = (self._scrub(exc.args[0]),) + exc.args[1:]
        return True

    def _by_key(self, key: str, value):
        return fingerprint(str(value)) if key.lower() in SECRET_KEYS else self._scrub(value)

    def _scrub(self, value):
        if not isinstance(value, str):
            return value
        for pattern in VALUE_PATTERNS:
            value = pattern.sub("[redacted]", value)
        return value

Field 2 is the one that surprises people. logger.info("auth failed for %s", token) never puts the token in record.msg — it sits untouched in record.args until the formatter interpolates it, long after a message-only filter has run and passed.

Step 3 — Attach it to the logger, not the handler. In dictConfig, filters listed on a logger run in Logger.handle() before any handler is consulted; filters listed on a handler run inside Handler.handle(), once per handler, after the record has already been passed around.

CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "filters": {"redact": {"()": "observability.redact.RedactFilter"}},
    "formatters": {"json": {"()": "pythonjsonlogger.jsonlogger.JsonFormatter"}},
    "handlers": {
        "queue": {"class": "logging.handlers.QueueHandler", "queue": "cfg://custom.queue"},
    },
    "root": {"level": "INFO", "handlers": ["queue"], "filters": ["redact"]},
}

One caveat that catches people: filters on a logger do not apply to records propagated up from child loggers — Logger.filter() runs only on the logger the call was made on. Attach the filter to the root logger and make sure library loggers propagate to it, or attach it to each top-level logger your code uses. Verifying that is a one-line assertion worth having in a test.

Step 4 — Choose mask or fingerprint per field. [redacted] for anything you never need to correlate. A salted fingerprint for values where "the same token failed forty times in a minute" is the signal you are looking for. Never a truncation — the first six characters of an API key are enough to identify the customer and sometimes enough to guess the rest.

Configuration options

Verification

Test the record, not the rendered line. A test that greps formatter output passes for the wrong reason the day someone changes the formatter.

import logging

def test_token_in_args_is_redacted():
    records = []
    logger = logging.getLogger("t")
    logger.addFilter(RedactFilter())
    logger.addHandler(type("H", (logging.Handler,), {"emit": lambda s, r: records.append(r)})())
    logger.setLevel(logging.INFO)

    logger.info("auth failed for %s", "Bearer sk-live-9f3c8a2b1d4e")
    assert "sk-live" not in records[0].args[0]        # the args tuple, not the message

def test_extra_key_is_fingerprinted():
    ...
    logger.info("login", extra={"session": "s_9f3c8a2b"})
    assert records[0].session.startswith("sha256:")

Expected Output:

test_redact.py::test_token_in_args_is_redacted PASSED
test_redact.py::test_extra_key_is_fingerprinted PASSED

Then confirm the end-to-end shape once, so you know the filter is actually installed in the running configuration rather than only in the test:

logging.config.dictConfig(CONFIG)
logging.getLogger("service").info(
    "connecting to %s", "postgresql://svc:hunter2@db.internal:5432/app",
    extra={"api_key": "sk-live-9f3c8a2b1d4e"},
)

Expected Output:

{"levelname": "INFO", "name": "service",
 "message": "connecting to postgresql://[redacted]db.internal:5432/app",
 "api_key": "sha256:1b9d4f0a77c2"}

Common mistakes

The filter is on the handler and one sink already wrote the record

Error signature: the JSON sink shows [redacted] and the file sink shows the secret. Root cause: handler filters run per handler, and the logger hands the same record to each in registration order. Remediation: move the filter to the logger. Keep handler filters for severity routing and other decisions that are genuinely per-sink.

Interpolated arguments slip through

Error signature: a secret appears in the rendered message although the filter clearly rewrites record.msg. Root cause: the value lives in record.args and is only merged into the message at format time. Remediation: rewrite record.args as well, handling both the tuple and the mapping form.

Redaction turns into a CPU cost on the hot path

Error signature: P99 latency rises after deploying the filter; profiles show time in re.sub. Root cause: a dozen patterns applied to every record at DEBUG volume, most of which cannot match the field being scanned. Remediation: anchor the patterns, apply value scanning only to string fields, and skip records below INFO on hot paths. The measurement technique is in measuring Python logging overhead.

Choosing what to redact

A redaction filter is only as good as its list, and the list is a policy decision rather than a technical one. Three questions decide it, and answering them once beats discovering the answer during an incident review.

What must never appear, anywhere? Credentials, tokens, private keys, full card numbers, and anything that would let a reader authenticate as someone else. These are unconditional: mask them regardless of level, environment or how convenient the value would be for debugging. This is the set that justifies a key-based rule rather than a pattern, because you can enumerate it.

What is personal data? Email addresses, names, addresses, phone numbers, and identifiers that resolve to a person. The right treatment here is usually a fingerprint rather than a mask, because support and debugging genuinely need to know that the same customer appeared in fifty records — they just do not need to know which customer from the log itself. A fingerprint plus a separate, access-controlled lookup satisfies both needs.

What is merely commercially sensitive? Internal hostnames, pricing rules, queue depths, feature flags. These usually do not need redaction in a log your own team reads, and do need it in anything exported to a vendor or a support ticket. The practical approach is to leave them in the record and handle them at the export boundary, because redacting them in-process removes information your own engineers need.

Where the list comes from

Deriving the key list from your own schema beats maintaining it by hand. If the application already has serialisation models — dataclasses, Pydantic models, an ORM — the fields marked sensitive there are the same fields that must be masked in logs, and generating the set at import time keeps the two from drifting.

from dataclasses import fields

def sensitive_keys(*models) -> frozenset[str]:
    """Every field annotated as sensitive across the given models."""
    found = set()
    for model in models:
        for f in fields(model):
            if f.metadata.get("sensitive"):
                found.add(f.name)
    return frozenset(found)

SECRET_KEYS = sensitive_keys(User, PaymentMethod, ApiCredential) | {"authorization", "cookie"}

A field added to a model with the sensitive marker is then covered by redaction the moment it exists, with no second change to remember.

Verifying the policy, not just the code

The test that matters is not "does the filter mask this string" but "can any known-sensitive value reach a sink". A property-style test that walks the model definitions and asserts that each sensitive field name is in the filter's key set catches the case the unit tests miss, which is a new field nobody thought about.

def test_every_sensitive_model_field_is_redacted():
    for model in (User, PaymentMethod, ApiCredential):
        for f in fields(model):
            if f.metadata.get("sensitive"):
                assert f.name in SECRET_KEYS, f"{model.__name__}.{f.name} is not redacted"

One caveat worth stating plainly: redaction is a safety net rather than a design. The strongest control is not logging the value in the first place, and a filter that has to mask a field on every record is a hint that the field should not have been passed to the logging call at all.

Related

• Exception and traceback logging in Python — the parent guide: how exception text reaches the record in the first place.
• Capturing unhandled exceptions and warnings — hooks whose records must pass through the same filter.
• Handler architecture for Python logging — why the queue boundary is the point of no return.
• Configuring logging with dictConfig — attaching filters at the logger in a declarative config.
• Measuring Python logging overhead — costing the filter before it ships.

Frequently Asked Questions