Adopting ECS Fields for Python Logs

Elastic Common Schema names log fields in a way that log stores, dashboards and detection rules already understand: log.level, service.name, error.stack_trace, event.outcome. A Python service emitting those names gets working dashboards and search without configuration; one emitting levelname and exc_info gets a store full of fields nothing recognises. This page covers mapping Python's LogRecord onto ECS, the fields that matter most, and where the mapping should live. It is a task article under JSON log schemas and conventions, part of the modern Python logging libraries deep dive section.

Prerequisites

pip install "ecs-logging>=2.1.0,<3.0.0" \
            "python-json-logger>=2.0.7,<4.0.0"

The ecs-logging package provides a ready-made formatter; the implementation below shows the mapping explicitly so it can be adapted.

Implementation

Step 1 — Map the record's own attributes. A LogRecord already carries everything needed for the core ECS fields: the creation time, the level name, the logger name, the formatted message, the source location, and process and thread identifiers. The mapping is mechanical, and doing it in the formatter means no call site ever needs to know about it.

import logging
from datetime import datetime, timezone

def ecs_core(record: logging.LogRecord) -> dict:
    return {
        "@timestamp": datetime.fromtimestamp(record.created, tz=timezone.utc)
                              .isoformat(timespec="milliseconds").replace("+00:00", "Z"),
        "log.level": record.levelname.lower(),
        "log.logger": record.name,
        "log.origin.file.name": record.filename,
        "log.origin.file.line": record.lineno,
        "log.origin.function": record.funcName,
        "message": record.getMessage(),
        "process.pid": record.process,
        "process.thread.id": record.thread,
        "ecs.version": "8.11.0",
    }

Step 2 — Add service identity from configuration. ECS expects service.name, service.version and service.environment on every record. None of them is known to the LogRecord; all of them are known at startup. Setting them once in the formatter's constructor, from environment variables, makes every record attributable without any call site passing them.

Step 3 — Put exceptions in the error object. ECS places exception details in error.type, error.message and error.stack_trace. Keeping the stack trace out of the message field means the message stays a short, groupable description, and a search on error.type finds every occurrence of an exception class directly.

def ecs_error(formatter: logging.Formatter, record: logging.LogRecord) -> dict:
    if not record.exc_info or not record.exc_info[0]:
        return {}
    exc_type, exc_value, _ = record.exc_info
    return {
        "error.type": f"{exc_type.__module__}.{exc_type.__qualname__}",
        "error.message": str(exc_value)[:1000],
        "error.stack_trace": formatter.formatException(record.exc_info),
    }

Step 4 — Record outcomes in the event fields. event.action, event.category and event.outcome are what security and audit tooling reads. A login attempt with event.category: authentication, event.action: user-login and event.outcome: failure is recognised by detection rules without any custom configuration. Populating them for the events that matter — authentication, authorisation, data access — is a small change with outsized benefit.

log.warning("login failed", extra={
    "event.category": "authentication",
    "event.action": "user-login",
    "event.outcome": "failure",
    "user.id": user_id,
    "source.ip": client_ip,
})

Step 5 — Namespace application fields. Anything that is not an ECS field goes under a single custom object. ECS reserves many top-level names, including some it has not yet used, and a service field that happens to share a name with a future ECS field would silently collide. A namespace named for the organisation avoids that permanently.

from pythonjsonlogger import jsonlogger

ECS_KEYS_ALLOWED_FROM_EXTRA = {
    "event.category", "event.action", "event.outcome", "user.id", "source.ip",
    "trace.id", "span.id", "transaction.id", "http.request.method",
    "http.response.status_code", "url.path", "event.duration",
}

class EcsFormatter(jsonlogger.JsonFormatter):
    def __init__(self, service: str, version: str, environment: str, namespace: str = "acme"):
        super().__init__()
        self._service = {"service.name": service, "service.version": version,
                         "service.environment": environment}
        self._ns = namespace

    def add_fields(self, target, record, message_dict):
        target.clear()
        target.update(ecs_core(record))
        target.update(self._service)
        target.update(ecs_error(self, record))
        custom = {}
        for key, value in record.__dict__.items():
            if key in ECS_KEYS_ALLOWED_FROM_EXTRA:
                target[key] = value
            elif key not in logging.LogRecord("", 0, "", 0, "", (), None).__dict__ \
                    and key not in ("message", "asctime"):
                custom[key] = value
        if custom:
            target[self._ns] = custom

Expected Output: a record that an ECS-aware store renders and searches without configuration.

{"@timestamp": "2026-09-18T14:02:11.408Z", "log.level": "warning", "log.logger": "auth", "message": "login failed", "service.name": "identity", "service.version": "2026.09.18", "service.environment": "production", "event.category": "authentication", "event.action": "user-login", "event.outcome": "failure", "user.id": "u_8f21c9", "source.ip": "203.0.113.7", "ecs.version": "8.11.0", "acme": {"attempt": 3}}

The fields that matter most

ECS defines hundreds of fields, and adopting all of them at once is neither necessary nor helpful. A small set delivers most of the value, and extending beyond it can happen as specific needs appear.

Identity: service.name, service.version, service.environment. Every query that spans services, every dashboard filtered by environment, and every comparison between releases depends on these. They cost nothing per call site because the formatter sets them.

Severity and origin: log.level, log.logger. The basis of every filter. Using the ECS names, in lowercase as ECS specifies, means the store's built-in level views work.

Errors: error.type, error.message, error.stack_trace. The fields an incident starts from. Separating them from the message keeps both searchable.

Correlation: trace.id, span.id, transaction.id. The join to tracing. ECS's names for these differ from the OpenTelemetry attribute names, which is one of the few places a fleet using both conventions needs a mapping. Populating them from the active span is covered in adding trace IDs to log records.

Outcomes: event.category, event.action, event.outcome. Only for the events that security and audit tooling consumes — authentication, authorisation, data access, configuration change. Most operational records do not need them.

HTTP: http.request.method, http.response.status_code, url.path. For access logs and request-scoped records. Using ECS names here makes request logs join naturally with the store's HTTP dashboards.

That is around fifteen fields. A service emitting them consistently, with everything else in its own namespace, is well served by an ECS-aware store, and further fields can be added when a specific dashboard or detection rule needs them.

Dotted names versus nested objects

ECS fields are written with dots, and there are two ways to represent them in JSON: as flat keys containing dots — "log.level": "warning" — or as nested objects — "log": {"level": "warning"}. Stores that understand ECS generally accept both and index them identically, and the choice matters mostly for consistency and for tooling that is not ECS-aware.

Nested objects are the canonical form in the specification and are what most ECS-aware tooling produces. They are also easier for generic JSON consumers to navigate, since a field such as error.stack_trace is reachable as a normal path rather than as a key that happens to contain dots. The cost is a slightly more complex formatter, because every dotted name has to be split and merged into the right object.

Flat dotted keys are simpler to produce and read naturally in raw output, which is why the examples above use them. They behave identically in an ECS-aware store. They can confuse generic tools that treat dots as path separators and then find no nested object, and they make it possible to accidentally emit both log.level as a flat key and a nested log object in the same record, which some stores reject.

The rule that avoids trouble is to pick one representation in the shared formatter and apply it to every field, including custom ones. Mixing representations within a fleet — or worse, within a record — produces the indexing surprises that a schema was meant to prevent. The ecs-logging package produces nested output, which is a reasonable default for fleets without a strong preference.

Configuration options

Verification

Validate a representative record against the ECS field set the store expects.

import json
record = json.loads(formatted_line)
required = {"@timestamp", "log.level", "log.logger", "message",
            "service.name", "service.environment", "ecs.version"}
missing = required - record.keys()
stray = [k for k in record if "." not in k and k not in required
         and k not in {"message", "acme"}]
print("missing:", sorted(missing) or "none")
print("non-ECS top-level:", stray or "none")

Expected Output:

missing: none
non-ECS top-level: none

A non-empty "non-ECS top-level" list names fields that escaped the namespace, which are the ones at risk of colliding with a standard field.

Common mistakes

Uppercase levels. Error signature: the store's level views showing nothing. Root cause: WARNING where ECS expects warning. Remediation: lowercase in the formatter.

Stack traces in the message. Error signature: messages that cannot be grouped and exception types that are not searchable. Root cause: the traceback appended to the message text. Remediation: the error object.

Custom fields at the top level. Error signature: a service field colliding with an ECS field added later. Root cause: no namespace. Remediation: everything non-ECS under one custom object.

Collector mapping made permanent. Error signature: a mapping that silently stops working after an application change. Root cause: the translation is not tested with the code that produces the records. Remediation: move it into the formatter once the service can be changed.

Adopting every field at once. Error signature: a long migration that stalls. Root cause: treating the whole specification as the target. Remediation: start with the fifteen fields above and extend by need.

Frequently Asked Questions