Python Observability

Shipping Python Logs with Fluent Bit

A Python service that writes one JSON object per line to standard output has done its half of the job. This page covers the other half: a Fluent Bit configuration that reads those lines out of the container runtime's files, recovers the application's own fields, attaches the pod metadata, and does not lose records when anything restarts. It is a task article under log shipping and collection, part of the Python telemetry pipelines and delivery section.

What each stage does to the record A single log line is followed through four stages. The Python process emits a JSON object. The container runtime wraps it in its own envelope containing a timestamp, a stream name and the original line as an escaped string, and writes that to a file. The Fluent Bit tail input parses the envelope, producing a record with the application's JSON still trapped inside a single text field. The merge step parses that inner payload and promotes its keys to top level, so level, logger and trace identifier become real fields. Finally the enrichment filter adds pod, namespace and node names taken from the file path and the cluster API. The diagram marks the merge step as the one that is off by default and the one whose absence produces the single most common complaint about this pipeline. one line, four transformations {"msg":"order ok","level":"INFO"} what Python wrote 2026-09-18T… stdout F {"msg":…} the runtime envelope on disk tail input + cri parser envelope parsed · payload still a string Merge_Log On off by default — this is the missing step kubernetes filter — pod, namespace, node, labels attached from the file path and the API
The application's JSON survives the runtime envelope only if the merge step is enabled. Everything else in this pipeline is plumbing.

Prerequisites

Fluent Bit runs as a DaemonSet on each node or as a sidecar beside one pod; in neither case is anything installed into the Python service, which is the property that makes this arrangement worth the extra component. The application side needs only a formatter that emits one JSON object per line, and the rest of this page is configuration that lives with the platform rather than with the code:

pip install "python-json-logger>=2.0.7,<4.0.0"

# the two facts the configuration depends on
ls /var/log/containers/            # the runtime's log directory
cat /etc/os-release | head -1      # the runtime, which selects the parser

Implementation

Step 1 — Tail the runtime's files with the matching parser. Container runtimes do not store the bytes your process wrote. They store an envelope: a timestamp assigned by the runtime, the stream name, a partial-line flag, and then your line as the payload. Selecting the wrong parser for that envelope produces records whose entire content is one unparsed string, and — this is the part that costs an afternoon — such records look almost correct in a log viewer, because the text is all there. The failure only becomes visible when somebody tries to filter on a field and gets nothing.

The DB path deserves a moment's attention even though it is one line. It stores the byte offset reached in each file. Without it the input starts at the end of every file it opens, so every restart of the collector silently discards whatever was written while it was down. With it, a restart resumes exactly where it stopped.

[SERVICE]
    Flush            5
    Daemon           Off
    Log_Level        info
    Parsers_File     parsers.conf
    HTTP_Server      On          # exposes the metrics used in verification below
    HTTP_Listen      0.0.0.0
    HTTP_Port        2020

[INPUT]
    Name             tail
    Tag              kube.*
    Path             /var/log/containers/*.log
    Parser           cri
    DB               /var/fluent-bit/state/flb.db
    Mem_Buf_Limit    32MB
    Skip_Long_Lines  Off
    Refresh_Interval 10

Step 2 — Merge the application's own JSON. This is the step whose absence accounts for most of the confusion about this pipeline. Without it, level, logger and trace_id are characters inside a string rather than fields you can filter on, and the store indexes that string as a single blob of text. With it, the payload is parsed a second time and its keys are promoted, so the fields the Python formatter carefully produced become the fields the query language sees.

The same filter is also the one that attaches pod metadata, which it derives partly from the log file's own path — the file name encodes pod, namespace and container — and partly from the cluster API for labels and annotations. That second source is why the filter needs a service account and why it fails in a way that looks like a metadata problem when the permission is missing.

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Merge_Log           On
    Merge_Log_Key       ""          # promote to top level, do not nest
    Keep_Log            Off         # drop the raw copy once parsed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

Step 3 — Normalise the fields you will query on. Different Python logging stacks spell the same idea differently: the standard library's JSON formatters emit levelname, structlog emits level, and a service that grew its own formatter may emit severity. A rename stage in the collector costs nothing at runtime and prevents the fleet from accumulating three spellings that every dashboard then has to account for. Doing it here rather than in each application also means a service can be brought into line without being redeployed.

[FILTER]
    Name          modify
    Match         kube.*
    Rename        levelname level
    Rename        asctime   timestamp
    Add           collector fluent-bit

Step 4 — Forward, with a bounded retry. The destination matters less here than the retry policy. Fluent Bit holds a batch in memory while it retries, and that memory counts against the same budget the input is reading into, so a forwarder configured to retry indefinitely against a backend that is down will eventually stop the input entirely. A bounded retry limit means one batch is abandoned after a few attempts, which is the correct trade for ordinary application logs and the wrong one for audit records — those belong on a path that persists to disk instead.

[OUTPUT]
    Name          es
    Match         kube.*
    Host          logs.observability.svc
    Port          9200
    Logstash_Format On
    Retry_Limit   5
    Suppress_Type_Name On

Expected Output: a record as it arrives at the store, with the application's fields at top level and the platform's alongside them.

{
  "@timestamp": "2026-09-18T09:14:02.881Z",
  "message": "order accepted",
  "level": "INFO",
  "logger": "orders",
  "trace_id": "9f2a71c4f0b84c2e9d5f1a7b3c8e6d02",
  "kubernetes": {
    "pod_name": "checkout-7d9f8c5b6-xk2lm",
    "namespace_name": "payments",
    "container_name": "app",
    "host": "ip-10-0-4-91"
  },
  "collector": "fluent-bit"
}
What the state database is protecting Two timelines of the same file being written continuously. In the upper timeline Fluent Bit has a state database on a persistent volume. It stops at one point, the file keeps growing for forty seconds, and on restart the stored offset is read so collection resumes exactly where it left off and the forty seconds of records are delivered late but completely. In the lower timeline there is no state database. The same restart occurs, and on start the input positions itself at the end of the file, so the forty seconds of records written during the gap are never read and are lost with no error anywhere. The note added is that the loss is invisible from the collector's own metrics, because from its point of view it successfully read everything it was ever asked to read. a restart, with and without a persisted offset DB on a volume collected agent down 40 s resumes at the stored offset — nothing missing no DB collected written, never read starts at the end of the file the collector reports complete success in both rows — it read everything it ever looked at only a per-service record rate, or a sequence number in the record, shows the difference
The state database is one line of configuration and the difference between a restart that costs nothing and one that silently removes a window of records.
Fluent Bit settings that matter for Python logs A table of Fluent Bit settings that most affect Python service logs. The tail input's Parser set to docker or cri unwraps the runtime's envelope. A second parser, json, decodes the Python record in the log field. Mem_Buf_Limit on the input bounds memory when the output is slow, pausing the input rather than growing. storage.type filesystem buffers to disk so records survive a restart. The DB option records file offsets so a restart resumes where it stopped. Retry_Limit on the output decides how long a failing destination is retried. The note says without Mem_Buf_Limit or filesystem storage, a slow destination grows Fluent Bit's memory until it is killed. setting effect Parser docker / cri unwraps the runtime envelope json parser on log field decodes the Python record Mem_Buf_Limit pauses input instead of growing storage.type filesystem buffered records survive restart DB (offset file) resume where it stopped Retry_Limit how long a failing output is retried without a memory bound or disk buffer, a slow destination grows memory until OOM
Two parsers make the record usable; the buffer and offset settings decide what survives a slow destination or a restart.

Configuration options

Key Section Recommended Effect
Parser INPUT cri or docker must match the runtime, or the payload stays a string
DB INPUT a path on a volume read position survives a restart
Mem_Buf_Limit INPUT 32MB pauses the input instead of growing memory
Skip_Long_Lines INPUT Off long records are counted, not silently dropped
Merge_Log FILTER On promotes the application's JSON keys to fields
Keep_Log FILTER Off removes the duplicated raw copy
Retry_Limit OUTPUT 5 bounds how long one batch can block the queue
Flush SERVICE 5 seconds trades delivery latency against request count

Operating it on a busy node

Two operational properties matter more than any single configuration key.

The first is that a node's log directory is shared. A DaemonSet reads every container's logs, so one extremely noisy pod consumes the collector's buffer and delays every other pod's records on that node. This is worth knowing before an incident, because the symptom — one service's logs arriving late — usually has nothing to do with that service. The remedy is either a per-pod rate limit at the source, as in rate limiting and sampling noisy loggers, or a sidecar for the pod that genuinely needs its own buffer.

The second is that the runtime rotates these files, and rotation is where a slow collector loses data permanently. The runtime keeps a small number of rotated files per container; if the collector is far enough behind when rotation happens, the file it was reading is unlinked and the remaining records go with it. The measurement that catches this is the lag between the newest record on disk and the newest record delivered, which is more useful than any absolute throughput number because it scales with the service.

Verification

Fluent Bit exposes its own counters on the HTTP port enabled in step 1. The two numbers that matter are records read and records that failed to reach the output.

curl -s localhost:2020/api/v1/metrics | python3 -m json.tool

Expected Output: input and output counts agree, and the error counters are flat.

{
  "input": { "tail.0": { "records": 1842301, "bytes": 704239112 } },
  "filter": { "kubernetes.0": { "add_records": 0, "drop_records": 0 } },
  "output": { "es.0": { "proc_records": 1842301, "errors": 0, "retries": 4, "retries_failed": 0 } }
}

Then prove the path end to end with a token that exists nowhere else, as in the parent guide:

import logging, uuid
probe = uuid.uuid4().hex
logging.getLogger("startup").info("fluent-bit probe", extra={"probe": probe})

Searching the store for that value should return exactly one document, carrying both the application fields and the pod metadata. If it returns a document whose log field contains the JSON as text, step 2 is not in effect.

Common mistakes

Every field is nested under log as a string. Error signature: a document with one long log value containing braces. Root cause: Merge_Log is off, or the payload is not valid JSON because a message contained an unescaped newline. Remediation: enable merging, and escape newlines in the formatter as described in log shipping and collection.

Records vanish across a Fluent Bit restart. Error signature: a gap in per-service record rate exactly as long as the restart. Root cause: no DB, or a DB path inside the container filesystem rather than on a volume. Remediation: put the state database on a mounted volume and confirm it survives a pod delete.

Pod metadata is missing but the application fields are present. Error signature: documents with level and logger but no kubernetes object. Root cause: the filter could not reach the cluster API, usually a missing service account permission or a network policy. Remediation: check the collector's own log at startup, where the API call failure is reported once and then never again, which is why it is so often missed.

Memory climbs until the pod is killed. Error signature: the collector container terminated for exceeding its memory limit while the destination was slow. Root cause: no Mem_Buf_Limit, so the input kept reading into memory faster than the output drained. Remediation: set the limit; the input then pauses and the records wait on disk in the runtime's files, which is where they are safest anyway.

Frequently Asked Questions

Why do my log fields end up inside a single string field?

The runtime wraps each line in its own envelope, and the application's JSON is the payload inside that envelope. Without log merging enabled, Fluent Bit parses the envelope and leaves the payload as text. Turning on Merge_Log parses the payload too, promoting its keys to top-level fields.

What happens to records written while Fluent Bit is restarting?

Nothing, provided the tail input has a persisted state database on a volume that outlives the process. On restart it reads the stored offset and continues. Without it, the input resumes at the end of the file and everything written during the gap is skipped.

Does Fluent Bit handle multi-line Python tracebacks?

It has a multiline parser that can join continuation lines, and the built-in python mode recognises the standard traceback shape. It is still better to emit the traceback as a JSON string field, because then no parser has to be correct for the record to survive.

How much memory does Fluent Bit need for a busy Python service?

The memory buffer limit is the number that decides it. Thirty-two megabytes per input is comfortable for a service producing a few thousand records per second; when the limit is reached the input pauses, which applies backpressure to the file rather than growing without bound.

Related Guides