JSON Log Schemas and Conventions for Python Services

Structured logging makes each record parseable. It does not make two services' records comparable, and a fleet where every team chose its own field names has structured logs that cannot be queried as a whole: duration_ms in one service, latency in another, elapsed as a string in a third. This guide covers the schema layer — naming, types, namespacing — and the two established conventions worth adopting instead of inventing one. It is part of the modern Python logging libraries deep dive section, and it pairs with the child pages on adopting ECS fields for Python logs, the OpenTelemetry log data model in Python and designing a log schema for a service fleet.

Prerequisites

pip install "python-json-logger>=2.0.7,<4.0.0" \
            "structlog>=24.1.0,<26.0.0" \
            "jsonschema>=4.21.0,<5.0.0"

Concept and architecture

A log schema has four parts, and fleets usually get the first right and the other three wrong.

Names. What each field is called. This is where most discussion happens and where conventions help most: adopting an existing set of names ends the discussion and makes the fleet's logs compatible with tooling built for that convention.

Types. Whether each field is a string, an integer, a float, a boolean or an object. Types matter more than names, because a store can alias two names but cannot reconcile two types — a field indexed as a number rejects records where it is a string. Type drift is the most common and most destructive schema failure, because the rejected records simply do not arrive.

Units. Milliseconds or seconds, bytes or kilobytes. A field called duration with no unit is an invitation for two services to disagree by a factor of a thousand. Encoding the unit in the name — duration_ms, size_bytes — removes the ambiguity at no cost.

Namespacing. Which fields are shared and which belong to one service. A shared field has a fleet-wide meaning; an application field has a meaning only within its service. Putting application fields under a namespace — an app object, or the service's own name — guarantees they can never collide with a shared field, which is the failure mode where a service's status field means something different from everyone else's.

The four parts also differ in how expensive they are to change later. Adding a field is free. Renaming one costs a transition period with both names emitted. Changing a unit costs every dashboard that plotted the old one. Changing a type costs a reindex or a new index, because stores cannot reinterpret stored data. That ordering is a good guide to where care should go at the start: types first, units second, names third.

Two established conventions cover the shared names well. The Elastic Common Schema is widely supported by log stores and defines names for common concepts in a dotted hierarchy. The OpenTelemetry log data model defines the structure of a log record alongside traces and metrics, with semantic conventions for attribute names shared across all three signals. They overlap substantially, and a fleet can use one in applications and map to the other at the collector.

Step-by-step implementation

Step 1 — Adopt a convention for the shared core. The core is small: timestamp, level, message, logger, service identity, environment, trace and span identifiers, error type and message, and a handful of HTTP fields. Taking those names from an established convention rather than debating them is the fastest route to agreement, and it means tooling built for that convention works immediately.

# the shared core — names from OpenTelemetry semantic conventions
CORE_FIELDS = {
    "@timestamp":            str,    # RFC 3339, UTC
    "level":                 str,    # DEBUG … CRITICAL
    "message":               str,
    "logger":                str,
    "service.name":          str,
    "service.version":       str,
    "deployment.environment": str,
    "trace_id":              str,    # 32 hex chars, or absent
    "span_id":               str,    # 16 hex chars, or absent
    "error.type":            str,
    "error.message":         str,
    "http.request.method":   str,
    "http.route":            str,
    "http.response.status_code": int,
    "duration_ms":           float,
}

Step 2 — Fix the type of every shared field, permanently. The list above assigns a type to each name, and the formatter enforces it. A value of the wrong type is coerced if unambiguous and dropped if not — a missing field is far less damaging than a field whose type conflicts with the store's mapping.

Step 3 — Namespace everything else. Service-specific data goes under a single object, so it can never collide with a shared name however it is spelled.

log.info("order accepted", extra={
    "duration_ms": 41.2,                            # shared, typed
    "app": {"order_id": "ord_7", "item_count": 3},  # this service's own data
})

Step 4 — Provide it as a package. A shared logging package that every service imports — containing the formatter, the core field constants and a helper for building records — makes compliance the path of least resistance. A schema that exists only as a document is followed by the services that read it, and drifts in the ones that do not.

# fleetlog/__init__.py — imported by every service
from .formatter import SchemaFormatter          # enforces names and types
from .fields import CORE_FIELDS, namespaced     # constants and the app-namespace helper
from .setup import configure                    # one call: formatter, handler, levels

Step 5 — Validate in continuous integration. Each service's snapshot tests, described in snapshot testing structured log output, produce representative records. Validating those against a JSON Schema generated from the core field list catches a renamed or retyped field in the pull request that introduces it.

import json
import jsonschema

SCHEMA = {
    "type": "object",
    "required": ["@timestamp", "level", "message", "service.name"],
    "properties": {
        name: {"type": {str: "string", int: "integer", float: "number"}[t]}
        for name, t in CORE_FIELDS.items()
    },
}

def test_snapshot_conforms(snapshot_record):
    jsonschema.validate(snapshot_record, SCHEMA)

Step 6 — Version the schema. Additions are compatible; renames and type changes are not. A version number in the shared package, and a changelog that says which kind each change is, lets services upgrade deliberately and lets dashboard owners know when a field they use is changing.

Configuration reference

Async and concurrency considerations

A schema is a property of the output, so concurrency affects it only through the fields that come from context.

Trace and span identifiers, request identifiers and tenant fields are usually populated from context variables at the moment the record is created. Under asyncio those are per task, and under threads per thread, which is correct as long as the formatter reads them when the record is created rather than when it is formatted. With a queue handler, formatting happens later on the listener thread, where the context variables have different values. The fix is a filter on the originating side that copies context into the record's attributes before it is enqueued, as described in adding trace IDs to log records; the schema formatter then reads the attributes, which travel with the record.

The second consideration is that a shared formatter runs on every record in every service, on whatever thread logs. Its schema enforcement — type checks, coercion, namespace handling — should be cheap and allocation-light. A formatter that validates every record against a full JSON Schema at runtime is correct and slow; the validation belongs in continuous integration, and the runtime formatter should enforce the core types with direct checks.

Production code examples

A formatter that enforces the core schema at the source — typed shared fields, a namespace for everything else, and drops rather than corrupts on an unfixable type:

# fleetlog/formatter.py
import logging
from datetime import datetime, timezone

from pythonjsonlogger import jsonlogger

from .fields import CORE_FIELDS

_RESERVED = set(logging.LogRecord("", 0, "", 0, "", (), None).__dict__) | {"message", "asctime"}


class SchemaFormatter(jsonlogger.JsonFormatter):
    def __init__(self, service: str, version: str, environment: str):
        super().__init__()
        self._static = {"service.name": service, "service.version": version,
                        "deployment.environment": environment}

    def add_fields(self, target, record, message_dict):
        target.clear()
        target["@timestamp"] = datetime.fromtimestamp(
            record.created, tz=timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
        target["level"] = record.levelname
        target["logger"] = record.name
        target["message"] = record.getMessage()
        target.update(self._static)

        app = {}
        for key, value in record.__dict__.items():
            if key in _RESERVED or key.startswith("_"):
                continue
            if key in CORE_FIELDS:
                coerced = _coerce(value, CORE_FIELDS[key])
                if coerced is not None:
                    target[key] = coerced             # 1. shared field, right type
            elif key == "app" and isinstance(value, dict):
                app.update(value)
            else:
                app[key] = value                      # 2. anything else is namespaced
        if app:
            target["app"] = app
        if record.exc_info:
            target["error.type"] = record.exc_info[0].__name__
            target["error.message"] = str(record.exc_info[1])[:500]
            target["error.stack_trace"] = self.formatException(record.exc_info)


def _coerce(value, expected):
    if isinstance(value, expected):
        return value
    try:
        return expected(value)                        # "41.2" -> 41.2, 7 -> "7"
    except (TypeError, ValueError):
        return None                                   # 3. absent beats conflicting

Expected Output: every service's records share a shape, and service-specific data is contained.

{"@timestamp": "2026-09-18T14:02:11.408Z", "level": "INFO", "logger": "orders", "message": "order accepted", "service.name": "orders", "service.version": "2026.09.18", "deployment.environment": "production", "trace_id": "9f2a71c4f0b84c2e9d5f1a7b3c8e6d02", "duration_ms": 41.2, "app": {"order_id": "ord_7", "item_count": 3}}

Choosing between the two conventions

Both established conventions are reasonable, and the choice between them depends mostly on where logs end up and what they travel alongside.

OpenTelemetry semantic conventions name attributes consistently across traces, metrics and logs. A log record carrying http.route and a span carrying http.route refer to the same thing, which makes joining signals straightforward and makes the conventions a natural fit when logs are exported through the OpenTelemetry logs pipeline. The conventions are organised around the resource that produced telemetry — service.name, deployment.environment, host.name — and the operation being described, and they are actively maintained as new domains are added. Their weakness for logs specifically is that they say less about log-only concepts such as the logger name or the source location, which fleets fill in themselves.

Elastic Common Schema was designed for logs and events first, and it has names for many things a log record carries that trace-oriented conventions do not emphasise: the log origin, the event category and outcome, user and client details. Log stores built around it provide dashboards and detection rules that work out of the box if the field names match. Its weakness is that trace and metric attributes follow different names, so joining signals needs a mapping.

In practice, the two have converged substantially and continue to, and the pragmatic arrangement for many fleets is to emit OpenTelemetry-style names from applications — so all three signals agree at the source — and to map to ECS names at the collector for any store or tooling that expects them. The mapping is a small, stable table, and keeping it in one place is far easier than asking every service to know about two conventions.

What matters far more than which convention is chosen is that one is chosen, early, and enforced. A fleet with a consistent home-grown schema is better off than one where half the services use ECS and half use their own names, and the cost of migration grows with every service that ships without a schema.

What does not belong in a schema

A schema can grow until it describes every field any service might want, and at that point it stops being useful. Three categories are better left out.

Business-specific fields. An order identifier, a product SKU, a tenant plan — these matter to one service or one domain and have no fleet-wide meaning. They belong in the application namespace, where each service can name them as it likes without negotiation. Promoting them to the shared schema creates a committee for decisions that only one team needs to make.

Derived values. A field that can be computed from others — a boolean is_error from the level, a slow flag from the duration — adds storage and the possibility of inconsistency. Queries can derive them; the schema should not.

Anything with unbounded variety. Free-text fields, stack traces and request bodies have no fixed shape to describe. The schema can name the field — error.stack_trace — and leave its contents unconstrained, which is different from trying to structure the contents themselves.

A useful test for a proposed shared field is whether a query that spans services would use it. If only one service's dashboards would, it belongs in that service's namespace.

Getting a fleet to adopt it

The technical work in this guide is small. The difficult part is moving an existing fleet, whose services each have their own established field names and dashboards built on them, onto a shared schema without breaking everything at once.

Start with the fields that cross services. Service identity, trace identifier, level and duration are used by nearly every cross-service query. Standardising those four first delivers most of the value — fleet-wide latency and error queries, and joining logs to traces — while touching the fewest dashboards.

Emit both names during the transition. A formatter can write the new field and the legacy one for a period, so existing dashboards keep working while new ones are built on the standard names. A date for removing the legacy fields, announced in advance, prevents the transition from becoming permanent.

Map at the collector for services that cannot change quickly. A collector rule that renames a legacy field to the standard name brings a service's output into line without a release, which is useful for services owned by teams with other priorities. It is a stopgap: the application should eventually emit the standard name itself, so that the rule can be removed and the application's own tests validate its output.

Make the shared package better than the alternative. Adoption follows convenience. A package that configures logging correctly in one call — JSON, UTC timestamps, trace correlation, queue handler, library noise suppression — is something teams want regardless of the schema, and the schema arrives with it.

Common mistakes

Inventing a convention. Error signature: months of discussion over field names. Root cause: starting from a blank page. Remediation: adopt OpenTelemetry semantics or ECS for the core and move on.

Names agreed, types not. Error signature: one service's records silently rejected by the store. Root cause: the same field as a string in one service and a number in another. Remediation: fix types per field and coerce in the formatter.

No namespace for application data. Error signature: a query on a shared field returning values with a different meaning from one service. Root cause: a service-specific field sharing a name with a shared one. Remediation: all application fields under app.

Units left implicit. Error signature: latency panels that are a thousand times off for one service. Root cause: duration in seconds here and milliseconds there. Remediation: the unit in the field name.

A schema that is only a document. Error signature: gradual drift in services that never read it. Root cause: no enforcement. Remediation: a shared formatter package and schema validation in CI.

Nested and dotted names mixed. Error signature: the same field appearing as service.name in some records and as service: {name: …} in others, indexed as two different fields. Root cause: some formatters flatten dotted keys and some nest them. Remediation: decide on one representation in the shared formatter and apply it to every record.

Validating at runtime. Error signature: logging becoming a measurable fraction of CPU. Root cause: full JSON Schema validation on every record. Remediation: cheap type checks at runtime, full validation in tests.

Frequently Asked Questions