Custom Log Record Factories and Extra Fields

Every record from this service should carry the service name, the version, and the deployment environment; every record from a request should carry the request ID. Neither belongs in the message string. This page covers the two mechanisms that put fields on a record — setLogRecordFactory for what is always true, extra and filters for what is sometimes true — and the formatter change that makes them appear in the output. It builds on formatter configuration, part of the Python logging fundamentals and structured data section.

Prerequisites

pip install "python-json-logger>=2.0.7,<4.0.0"

export SERVICE_NAME=checkout-api
export SERVICE_VERSION=2026.8.1
export DEPLOY_ENV=production

Implementation

Step 1 — Install a factory for the constants. logging.setLogRecordFactory() replaces the callable that builds every LogRecord. Wrap the existing one rather than replacing it outright, so another library's factory — an OpenTelemetry bridge, for instance — keeps working.

import logging
import os

_BASE_FACTORY = logging.getLogRecordFactory()

_SERVICE_FIELDS = {
    "service": os.environ.get("SERVICE_NAME", "unknown"),
    "version": os.environ.get("SERVICE_VERSION", "0"),
    "env": os.environ.get("DEPLOY_ENV", "dev"),
}

def record_factory(*args, **kwargs) -> logging.LogRecord:
    record = _BASE_FACTORY(*args, **kwargs)
    for key, value in _SERVICE_FIELDS.items():
        setattr(record, key, value)
    return record

logging.setLogRecordFactory(record_factory)

Keep the factory boring. It runs for every record the process creates — including records that a level check is about to discard — so it must not do I/O, walk a stack, or read anything expensive. Constants resolved once at import, assigned by attribute, is the whole budget.

Step 2 — Use extra for per-call fields, and namespace them. Keys in extra become attributes on the record, and logging refuses to overwrite the ones it owns: message, asctime, name, levelname, module, args, exc_info and the rest. A collision raises KeyError at the call site.

logger.info("order accepted", extra={"order_id": order.id, "items": len(order.items)})

# raises KeyError: "Attempt to overwrite 'module' in LogRecord"
logger.info("import finished", extra={"module": "orders"})

There is no flag to relax the check, and there should not be — a record whose levelname is a customer's name would break every downstream consumer. Prefix your fields (app_module) or pick a different name.

Step 3 — Read request context in a filter, not the factory. A request ID exists only inside a request. Putting it in the factory means every library record and every startup record carries request_id=None, which is noise in every index. A filter attached to the logger can check first.

import contextvars

request_id_var: contextvars.ContextVar[str | None] = contextvars.ContextVar("request_id", default=None)

class RequestContextFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        rid = request_id_var.get()
        if rid is not None:                          # omit the field rather than emit a null
            record.request_id = rid
        return True

The contextvar plumbing — setting it per request and keeping it across await boundaries and thread hand-offs — is covered in using contextvars for request tracing.

Step 4 — Make the formatter emit fields it was never told about. A record does not mark which attributes are yours. Build a reference set once from a throwaway record and emit the difference; the formatter then picks up any field added later without a code change.

import json

_RESERVED = set(
    logging.LogRecord("", 0, "", 0, "", (), None).__dict__
) | {"message", "asctime", "taskName"}

class FieldsFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        for key, value in record.__dict__.items():
            if key not in _RESERVED:                 # anything the factory, a filter, or extra added
                payload[key] = value
        if record.exc_info:
            payload["exception"] = self.formatException(record.exc_info)
        return json.dumps(payload, default=str)

Configuration options

Verification

import io, json, logging

def test_factory_and_extra_both_land():
    stream = io.StringIO()
    h = logging.StreamHandler(stream); h.setFormatter(FieldsFormatter())
    log = logging.getLogger("t"); log.addHandler(h); log.setLevel(logging.INFO)
    log.addFilter(RequestContextFilter())

    request_id_var.set("r-9f3c")
    log.info("order accepted", extra={"order_id": 8812, "items": 3})

    payload = json.loads(stream.getvalue())
    assert payload["service"] == "checkout-api"     # from the factory
    assert payload["request_id"] == "r-9f3c"        # from the filter
    assert payload["order_id"] == 8812              # from extra
    assert "args" not in payload                    # reserved fields stay out

Expected Output:

{"ts": "2026-08-02T12:14:51+0000", "level": "INFO", "logger": "t",
 "message": "order accepted", "service": "checkout-api", "version": "2026.8.1",
 "env": "production", "request_id": "r-9f3c", "order_id": 8812, "items": 3}

Add a second test that no field name collides with a reserved attribute — that failure surfaces at the call site, in production, on the one code path nobody exercised.

Common mistakes

KeyError: "Attempt to overwrite 'module' in LogRecord"

Error signature: the exception is raised by the logging call itself, not by a handler. Root cause: an extra key matches an attribute logging owns. Remediation: namespace the field. Keep a test that logs one record with every field name your codebase uses.

Every library record carries request_id: null

Error signature: the index is full of null request IDs from records that never had a request. Root cause: the request ID was added in the record factory, which runs for every record in the process. Remediation: move it to a filter that checks the contextvar and omits the attribute when there is nothing to set.

A new field never appears in the output

Error signature: the attribute is on the record in a debugger, but the JSON does not contain it. Root cause: the formatter emits an explicit allowlist of fields. Remediation: switch to the reference-set difference so any added field flows through automatically.

Designing the field set

Fields are cheap to add and expensive to remove, because a dashboard or an alert may come to depend on any of them. A small amount of design up front avoids a record schema that nobody can change.

Namespace by origin. Fields that describe the deployment (service, version, env, region), fields that describe the request (request_id, trace_id, tenant), and fields that describe the event (order_id, rows, duration_ms) come from three different mechanisms and change on three different schedules. Keeping them distinguishable — by prefix, or simply by documenting which is which — means a consumer knows which fields are guaranteed present. Deployment fields are always there; request fields are there inside a request; event fields are specific to one call site.

Fix the types. A field that is an integer in one call site and a string in another will be indexed as a string by most backends, and the numeric queries you wanted will not work. duration_ms as a float everywhere, status as an integer everywhere, order_id as whichever of the two your system actually uses — decided once. The formatter's default=str protects against a crash on a non-serialisable value, and it also silently turns a Decimal into a string, which is the sort of thing worth catching in a test rather than in a dashboard.

Keep names stable and short. Every field name is repeated in every record, so a verbose scheme costs real bytes at volume: svc versus service_name is nine bytes per record, which at 40 000 records per second is 31 MB an hour. That is not a reason to use cryptic names, but it is a reason not to be decorative.

Documenting the schema in code

The most useful thing you can do for the people querying these records is to make the field set discoverable without reading every call site. A module-level declaration serves that purpose and doubles as the thing your tests assert against.

# observability/schema.py
DEPLOYMENT_FIELDS = ("service", "version", "env")
REQUEST_FIELDS = ("request_id", "trace_id", "span_id")

def test_every_record_carries_the_deployment_fields(caplog):
    logging.getLogger("anything").info("probe")
    payload = json.loads(caplog.text)
    for field in DEPLOYMENT_FIELDS:
        assert field in payload

The test is worth more than it looks. The deployment fields come from a record factory that another library can replace — an instrumentation package, a vendor SDK, a testing tool — and when that happens the fields disappear silently from every record in the process. A single assertion catches it at build time instead of during the incident where somebody notices the version field has been missing for a month.

The interaction with third-party records

A record factory applies to every record in the process, including those from dependencies, which is exactly what you want for deployment identity: a urllib3 warning that carries your service name and version is more useful than one that does not. The corollary is that any field the factory adds must make sense on a record your code did not create. service does. order_id does not, which is why it belongs in extra at the call site rather than in the factory — and why a factory that reads request state produces a stream full of nulls, as described above.

Related

• Formatter configuration for Python logging — the parent guide: formatters, record fields, and output shapes.
• Structured logging with the Python standard library — the JSON formatter this page extends.
• Adding trace IDs to log records — the same filter mechanism, carrying trace context.
• Using contextvars for request tracing — where the request-scoped values come from.
• Binding context variables in structlog — the same idea with a library that treats fields as first class.

Frequently Asked Questions