Logging Personal Data Safely in Python

Personal data reaches logs by accident far more often than by design: a user object serialised for convenience, a request body logged while debugging, an exception message that includes an email address. Once there, it is copied to disks, collectors, backups and indexes, retained for as long as the logs are, and readable by everyone with log access. This page covers keeping it out by construction — identifiers instead of attributes, allow-lists instead of deny-lists, pseudonymisation where correlation matters — and why that also makes access and deletion requests answerable. It is a task article under logging security and compliance, part of the Python logging fundamentals and structured data section.

Prerequisites

pip install "python-json-logger>=2.0.7,<4.0.0"

# the key for pseudonymisation lives in a secret, never in code
export LOG_PSEUDONYM_KEY="$(cat /run/secrets/log-pseudonym-key)"

Implementation

Step 1 — Log identifiers, not attributes. The default for anything describing a person should be an opaque identifier that can be resolved to the person only through the system of record. An engineer who needs to know who a user is can look them up, under that system's access controls and audit trail. The log store, which is typically readable by far more people, never holds the attributes at all.

log.info("account updated", extra={"user.id": user.id, "fields_changed": ["address"]})

Note that the record says which fields changed without saying what they changed to. That is usually sufficient for operational purposes, and it is the pattern worth defaulting to.

Step 2 — Provide an allow-list helper for every sensitive type. Logging a whole object is the most common route for personal data into logs, and it happens because it is convenient. A helper per type that returns only the fields known to be safe makes the safe path equally convenient. Because it is an allow-list, a new field added to the model is excluded until somebody decides otherwise — it fails closed.

def log_fields_for_user(user) -> dict:
    return {
        "user.id": user.id,
        "user.tier": user.tier,
        "user.region": user.region,          # coarse, not the address
        "user.account_age_days": (today() - user.created).days,
    }

def log_fields_for_order(order) -> dict:
    return {
        "order.id": order.id,
        "order.item_count": len(order.items),
        "order.total_cents": order.total_cents,
        "user.id": order.user_id,
    }

log.info("order placed", extra=log_fields_for_order(order))

Step 3 — Pseudonymise where correlation is needed. Some investigations need to know that several records involve the same email address, IP address or device — to spot credential stuffing, for example — without needing the value itself. A keyed hash gives that: equal inputs produce equal outputs, and the original cannot be recovered without the key. The key must be secret and stable; rotating it breaks correlation across the rotation, which is sometimes desirable as a retention mechanism in itself.

import hashlib
import hmac
import os

_KEY = os.environ["LOG_PSEUDONYM_KEY"].encode()

def pseudonym(value: str, kind: str) -> str:
    digest = hmac.new(_KEY, f"{kind}:{value.strip().lower()}".encode(), hashlib.sha256)
    return f"{kind}_{digest.hexdigest()[:16]}"

log.warning("login failed", extra={
    "email.pseudonym": pseudonym(submitted_email, "email"),
    "ip.pseudonym": pseudonym(client_ip, "ip"),
})

Expected Output: records that can be grouped by attacker or target without containing either.

{"message": "login failed", "email.pseudonym": "email_3fa9c1d27be04e11", "ip.pseudonym": "ip_91d0a4e8f2c37b56"}
{"message": "login failed", "email.pseudonym": "email_3fa9c1d27be04e11", "ip.pseudonym": "ip_4c7e21b0d98a3f15"}

Step 4 — Add a pattern filter as a safety net. Allow-lists prevent the common case; free-text fields, exception messages and third-party library output still carry personal data occasionally. A filter that recognises the obvious patterns — email addresses, phone numbers, card numbers — on every string field catches most of what gets through. It is a net, not a wall: patterns miss things, and relying on the filter as the primary control is how personal data ends up in logs anyway.

import re

_EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
_PHONE = re.compile(r"\+?\d[\d\s().-]{8,}\d")

class PersonalDataFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        if isinstance(record.msg, str):
            record.msg = _PHONE.sub("[PHONE]", _EMAIL.sub("[EMAIL]", record.msg))
        for key, value in list(record.__dict__.items()):
            if isinstance(value, str) and key not in ("name", "levelname", "pathname"):
                setattr(record, key, _PHONE.sub("[PHONE]", _EMAIL.sub("[EMAIL]", value)))
        return True

Step 5 — Tag records that may contain personal data. Some records legitimately carry it — a support tool logging a lookup, an audit event naming a subject. Tagging those with a field such as data.personal: true lets the store route them to a destination with shorter retention and narrower access, rather than applying the strictest rules to everything or the loosest to these.

Deciding what is necessary

The hardest part of this in practice is not the mechanism but the judgement: what does an engineer actually need in a log record to do their job? The answer is usually much less than what gets logged, and three questions help establish it.

Would an identifier do? For almost every operational purpose — following a request, reproducing a bug, answering a support ticket — an identifier plus the ability to look it up is enough. The lookup adds a step, and that step is where access control and audit trails live. It is a feature, not an inconvenience.

Is the value itself the diagnostic, or its properties? A validation failure on an email address rarely needs the address; it needs to know which rule failed — missing domain, invalid characters, too long. Logging the property rather than the value answers the operational question without holding the data.

Who will read this? Logs are typically readable by every engineer on a team, often by support staff, sometimes by contractors. A value that would be inappropriate to show all of them should not be in a record they can all read. This framing tends to produce better decisions than abstract classification, because it makes the audience concrete.

When the answer is that a value genuinely is needed — a support tool that must record which customer a lookup concerned, for instance — that is the case for tagging the record, routing it to a restricted destination with short retention, and treating it as the exception it is. The goal is not zero personal data in any log ever; it is that every instance is deliberate, justified and handled accordingly. The retention side is covered in log retention and tiering strategy.

Traces and metrics carry personal data too

The discussion so far has been about log records, and the same risk exists in the other two signals, often less visibly.

Span attributes. Automatic instrumentation records request URLs, query strings, database statements and sometimes headers. A URL containing an email address as a query parameter, or a statement with a literal value rather than a placeholder, puts personal data into every span for that request, and spans are frequently retained and shared more broadly than logs. The allow-list and pseudonymisation techniques apply unchanged; the difference is that they have to be applied to instrumentation configuration — which attributes are captured, how URLs are sanitised — rather than to call sites.

Baggage. Values placed in baggage propagate to every downstream service, and often to external ones through outbound HTTP headers. A user's email address in baggage is sent to every service the request touches, including third parties. Baggage should carry identifiers and coarse attributes only, for exactly the reasons above, multiplied by the number of hops.

Metric labels. Personal data in a metric label is both a privacy problem and a cardinality problem, and the second usually surfaces first. An email address or user identifier as a label creates a series per person, which will be noticed on the metrics bill before it is noticed in a privacy review. The fix is the same as for cardinality generally — bounded labels only — as described in controlling label cardinality in Prometheus.

Configuration options

Verification

Search a sample of recent production records for the patterns that should never appear, which checks the controls against reality rather than against tests.

# count records in the last day containing email-shaped strings, per service
curl -sG 'http://logs:9200/logs-python-*/_count' \
  --data-urlencode 'q=message:/.*@.*\..*/ AND @timestamp:>now-1d'

Expected Output: zero, or a small number attributable to tagged records in the restricted destination.

{"count": 0}

A non-zero count outside the restricted destination names a leak worth tracing to its source — usually a third-party library's log output or an exception message.

Common mistakes

Logging whole objects. Error signature: personal attributes throughout the log store. Root cause: serialising a model for convenience. Remediation: an allow-list helper per type.

A deny-list of sensitive fields. Error signature: a new field leaking from the release that added it. Root cause: the default is to include. Remediation: switch to an allow-list.

Unkeyed hashes. Error signature: pseudonyms reversible by hashing a list of known emails. Root cause: a plain hash of a guessable value. Remediation: a keyed hash with a secret key.

Relying on the pattern filter. Error signature: personal data in formats the patterns do not recognise. Root cause: the net treated as the wall. Remediation: prevent at the source; keep the filter as a second layer.

Logging the value to explain a validation failure. Error signature: rejected personal data preserved in logs indefinitely. Root cause: the value logged rather than the rule it failed. Remediation: log which rule failed.

Frequently Asked Questions