Sending Python Logs to Loki

Loki is cheap because it indexes almost nothing: labels get an index, the log line does not. That single design decision is why a Loki deployment fed by a Python service either works beautifully or becomes unqueryable, and the difference is entirely in which fields became labels. This page covers label design for Python services, what belongs in the line instead, and how to tell that the stream count is the problem. It is a task article under log shipping and collection, part of the Python telemetry pipelines and delivery section.

Prerequisites

Nothing Loki-specific goes into the Python service. What it needs is JSON on standard output, which is the same prerequisite as every other collection path:

pip install "python-json-logger>=2.0.7,<4.0.0"

The collection agent — Promtail, Grafana Alloy, Fluent Bit with a Loki output — runs on the node and owns everything else.

Implementation

Step 1 — Decide which fields are labels. This is the whole design. A label is correct when its values are few, known in advance, and useful for selecting a group of logs: service name, namespace, environment, container, level. A label is wrong when its values are unbounded: trace identifier, user identifier, request path with parameters substituted in, or any error message. The test is simple — if you cannot write down the complete list of values it will ever take, it is not a label.

Level is a borderline case worth deciding deliberately. As a label it makes {level="ERROR"} instant, at the cost of multiplying every service's stream count by five or six. For most deployments that is a good trade, because the values are genuinely bounded and error-only queries are common.

Step 2 — Emit the rest as JSON inside the line. Everything that is not a label still has to be queryable, and in Loki that means it has to be parseable from the line at query time. One JSON object per line makes that a built-in operation rather than a regular expression somebody maintains.

import logging
from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter(
    "%(asctime)s %(levelname)s %(name)s %(message)s",
    rename_fields={"levelname": "level", "name": "logger"},
))
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)

logging.getLogger("orders").info(
    "order accepted",
    extra={"order_id": "ord_8812", "trace_id": "9f2a71c4f0b84c2e", "duration_ms": 41.2},
)

Expected Output: one line, with three fields that are queryable without being labels.

{"asctime": "2026-09-18T09:14:02.881Z", "level": "INFO", "logger": "orders", "message": "order accepted", "order_id": "ord_8812", "trace_id": "9f2a71c4f0b84c2e", "duration_ms": 41.2}

Step 3 — Let the agent attach the platform labels. The application never sets pod or namespace, for the same reason it never sets them in any other pipeline: those are facts about where it is running, and the agent already knows them. A relabelling stage keeps the set small and the names consistent.

# promtail scrape config — the labels, and nothing else
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs: [{ role: pod }]
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: service
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container
      # Deliberately NOT pod name: it changes on every deploy and multiplies streams.
      - action: labeldrop
        regex: __meta_.*
    pipeline_stages:
      - cri: {}
      - json:
          expressions: { level: level }
      - labels:
          level:

Step 4 — Query by filtering the line, not by adding labels. A field kept inside the line is still fast to query, because Loki reads only the streams the label selector matched and then scans those. The scan is linear in the bytes of the matched streams, which is why keeping stream count low is what makes line filtering cheap.

{service="checkout", namespace="payments"}
  | json
  | trace_id = "9f2a71c4f0b84c2e"

Step 5 — Watch active streams, not ingested bytes. Loki's volume limits are generous and its stream limits are not. A per-tenant active stream count that climbs with traffic rather than with pod count means a label is carrying an identifier, and it will be a query problem long before it is a storage problem.

Why the index design changes how you write logs

It is worth stating the consequence plainly, because it inverts an instinct carried over from search-based log stores. In a system that indexes content, adding detail to a record costs storage and makes queries better. In Loki, adding detail to the line costs storage and makes queries no worse; adding detail to the labels costs index size, stream count, memory and query latency, all superlinearly.

This has three practical effects on a Python service's logging.

First, verbosity inside the line is nearly free. A record carrying fifteen contextual fields is not meaningfully more expensive to query than one carrying three, because both are read by the same sequential scan of the same stream. The usual instinct to trim fields for cost reasons is aimed at the wrong number.

Second, the fields most worth carrying are the ones that let a query narrow after the label selector has done its work — the trace identifier, the tenant, the route, the outcome. These would be terrible labels and are excellent line fields.

Third, the structure of the line matters more than in an indexed store, because parsing happens at query time on every read. A line that is valid JSON parses with one operator. A line that requires a regular expression to pick apart costs that expression on every matched byte, every time anybody runs the query, forever. This is the strongest practical argument for structured logging in a Loki deployment specifically: the cost of unstructured lines is paid by every future query rather than once at write time.

One more consequence is worth naming because it affects retention planning. Loki stores chunks per stream, and a stream that receives very few records still produces chunks on a schedule. A deployment with a large number of near-idle streams therefore writes a large number of small objects to its backing store, which costs request charges rather than storage charges and is invisible on any volume graph. Consolidating labels reduces object count as well as query latency, and on an object store billed per request that second saving is frequently the larger one.

Configuration options

Verification

Check the stream count directly rather than inferring it from query speed.

# active streams for this tenant, and the label names in use
curl -s 'http://loki:3100/loki/api/v1/labels'
curl -s 'http://loki:3100/metrics' | grep -E '^loki_ingester_memory_streams'

Expected Output: a stream count proportional to services times containers times levels, and a label list with no identifiers in it.

{"status":"success","data":["container","level","namespace","service"]}
loki_ingester_memory_streams{tenant="default"} 1184

Eleven hundred streams for a cluster of forty services is healthy. If that number tracks request rate rather than pod count, a label is carrying an identifier and the label list above will name it.

Common mistakes

Queries time out while ingest volume is modest. Error signature: context deadline exceeded on queries over a few hours. Root cause: stream cardinality, usually from a pod-name or identifier label. Remediation: drop the label with a labeldrop rule in the agent; existing streams age out with retention.

A field is in the line but queries cannot see it. Error signature: | json returns no extracted fields. Root cause: the line is not valid JSON, often because a message contained a raw newline or the runtime envelope was not stripped first. Remediation: add the cri stage before the json stage, and escape newlines in the formatter.

Error-only dashboards are slow. Error signature: {service="x"} |= "ERROR" scanning everything. Root cause: level is being filtered from the line rather than selected by label. Remediation: promote level to a label in the agent's pipeline, which is the one place the multiplication is worth it.

Old logs cannot be queried after a label change. Error signature: a query that works for recent data returns nothing beyond a certain point in the past. Root cause: the label set changed, so older data lives in streams with different label combinations and the current selector does not match them. Remediation: query the old period with the old selector, and accept that a label change is effectively a schema change with a cutover date.

The agent is dropping lines under load. Error signature: entry out of order or rate-limit rejections in the agent log. Root cause: per-stream rate limits, hit because one stream carries a whole node's output. Remediation: raise the limit deliberately, or split the stream with a label that genuinely partitions the traffic — never with an identifier.

Frequently Asked Questions