Python Observability

Running the Collector as a Kubernetes Sidecar

A sidecar collector gives one pod a dedicated telemetry buffer and its own processor configuration, at the cost of a container per replica. This page covers when that trade is worth taking, how to wire the containers so startup and shutdown order work in your favour, and the ordering mistake that quietly loses the telemetry from every pod termination. It is a task article under collector topology and deployment, part of the Python telemetry pipelines and delivery section.

Termination order decides whether the last export lands Two termination sequences for the same pod. In the first, the collector runs as an ordinary container, so both containers receive the termination signal at the same moment and there is no ordering between them. The collector frequently exits first, which means the application's shutdown flush — the export carrying the spans and log records describing the shutdown itself — has nowhere to go and fails. In the second, the collector runs as a native sidecar, declared as an init container with an always restart policy. Kubernetes terminates the application container first, waits for it, and only then terminates the sidecar. The application's flush therefore reaches a collector that is still accepting, and the collector then flushes its own queue before exiting. The sequence marks the window in each case during which an export can still succeed. SIGTERM arrives — what happens next ordinary container app draining collector up collector exited — the app's flush has nowhere to go flush fails here native sidecar app draining, then flushes and exits collector still accepting collector drains, exits flush succeeds the records lost in the upper row are the ones describing why the pod went away
The telemetry produced during shutdown is the telemetry about shutdown. Container ordering is what decides whether it survives.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

The cluster must be running a Kubernetes version with native sidecar support, which is what provides the ordering the rest of this page depends on.

Implementation

Step 1 — Declare the collector as a native sidecar. A native sidecar is an init container with restartPolicy: Always. That one field changes three behaviours: the container starts and becomes ready before the application container starts, it is restarted independently if it crashes rather than failing the pod, and it is terminated after the application container during shutdown. All three matter here, and the last one is the reason to bother.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      initContainers:
        - name: otel-sidecar
          image: otel/opentelemetry-collector-contrib:0.109.0
          restartPolicy: Always          # <- this makes it a native sidecar
          args: ["--config=/conf/collector.yaml"]
          resources:
            requests: { memory: 128Mi, cpu: 50m }
            limits:   { memory: 256Mi, cpu: 500m }
          volumeMounts:
            - { name: collector-config, mountPath: /conf }
      containers:
        - name: app
          image: registry.example/checkout:2026.09.18
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://127.0.0.1:4317"
            - name: OTEL_SERVICE_NAME
              value: "checkout"

Step 2 — Export to the loopback address. Containers in a pod share a network namespace, so 127.0.0.1 reaches the sidecar and nothing else. This is the strongest form of the local-endpoint property: there is no node network involved, no service resolution, and no way for the hop to be affected by anything outside the pod. It also means the sidecar needs no authentication, because nothing other than this pod can reach it.

Step 3 — Give the sidecar a memory budget and hold it to it. The sidecar's queue is the buffer that justifies its existence, so it needs enough memory to hold something useful — but it shares the pod's memory accounting, and a collector that grows without bound will cause the pod to be evicted, taking the application with it. The memory limiter set below the container limit keeps the collector refusing rather than growing.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 192            # container limit is 256Mi
    spike_limit_mib: 48
  batch:
    timeout: 5s
    send_batch_size: 512
exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    sending_queue: { enabled: true, queue_size: 3000 }
    retry_on_failure: { enabled: true, max_elapsed_time: 120s }
service:
  telemetry:
    metrics: { address: 0.0.0.0:8888 }
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/gateway] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/gateway] }
    logs:    { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/gateway] }

Step 4 — Make the application flush before it exits. The sidecar ordering gives the application a window in which the collector is still accepting; the application has to use it. That means a termination handler that shuts the providers down explicitly, and a terminationGracePeriodSeconds long enough for the drain plus the flush. Without the handler the ordering buys nothing, because the process exits with its queue intact and unexported.

import signal
from opentelemetry import trace, metrics

def _drain(signum, frame):
    # 1. Stop accepting work first (framework-specific), then flush telemetry.
    trace.get_tracer_provider().shutdown()
    metrics.get_meter_provider().shutdown()

signal.signal(signal.SIGTERM, _drain)

Expected Output: during a rolling update, the last spans of the terminating pod arrive rather than disappearing.

2026-09-18T11:04:19Z  app       draining: 4 in-flight requests
2026-09-18T11:04:21Z  app       telemetry flushed: 318 spans, 12 metric points
2026-09-18T11:04:21Z  sidecar   exporter sent 318 spans
2026-09-18T11:04:22Z  sidecar   shutting down, queue empty

Step 5 — Apply it selectively. A sidecar is a per-service decision, and the default should remain the shared node collector. The services that earn one are those whose telemetry volume would otherwise dominate a node's shared buffer, those whose data must not transit a process another team configures, and those needing a processor that would be wrong to impose on everyone. Everything else is better served by the node agent, for the memory reasons set out in collector topology and deployment.

What the isolation costs Two lines plotted against the number of pod replicas on a node. The sidecar line rises linearly: every replica adds its own collector process with a memory floor of roughly eighty megabytes before any queue, so thirty replicas cost around two and a half gigabytes of memory that is idle most of the time. The shared node collector line is flat: one process serves every pod on the node, so its cost is the same whether the node runs three pods or thirty. The crossing point is at two replicas, which is to say the shared collector is cheaper almost immediately. Beside the lines is the property the sidecar buys in exchange, which is that one pod's telemetry burst cannot consume buffer belonging to a pod owned by another team. memory spent on collectors, per node pods on the node sidecar — one floor per replica node agent — one floor, shared MiB what it buys a burst from one pod cannot take buffer from another team worth it for two or three services, not forty
The shared collector is cheaper from the second replica onward. The sidecar buys isolation, and isolation is worth paying for in a small number of places.

What the pod boundary changes

Running the collector inside the pod changes more than the network hop, and three of those changes are worth understanding before adopting the pattern widely.

The collector's failures become the pod's failures. A node agent that crashes affects every pod's telemetry and none of their availability. A sidecar that exceeds its memory limit is a container termination inside your pod, and depending on how the deployment is configured that may restart the pod, fail a readiness probe, or simply appear as a restart count that somebody investigates as an application problem. The memory limiter matters more here than anywhere else, because its job is to prevent exactly this.

The configuration lifecycle changes owner. A node agent's configuration is platform-owned and changes rarely. A sidecar's configuration lives in the service's own manifests, which means the service team can change it — a genuine benefit when the point is a bespoke processor — and also that it drifts. Five services with sidecars will have five subtly different configurations within a year unless the base is shared through a template or an operator.

Resource accounting becomes honest. The sidecar's memory and CPU are charged to the pod, so a service's cost figure includes the cost of shipping its telemetry. With a node agent, that cost is a platform overhead that appears in nobody's service budget. This is a real advantage for organisations that allocate infrastructure cost to teams, and it occasionally makes the sidecar worth choosing for accounting reasons alone — a telemetry volume that is invisible tends to grow, and one that appears on a team's own bill tends not to.

There is also a subtler effect on scaling. A node agent's capacity is shared and therefore absorbs uneven load well: one pod's burst borrows headroom from its quiet neighbours. A sidecar has no neighbours, so it must be sized for that pod's peak rather than the node's average. In aggregate this means sidecars need more total memory than the node agent they replace, not less, which is the opposite of the intuition that a per-pod component must be smaller.

Sidecar resources that work A table of starting resource settings for a collector sidecar in a pod running one Python service. CPU request of 50 millicores and a limit of 200 millicores covers typical batching and export. Memory request of 64 mebibytes and a limit of 128 mebibytes, with the memory limiter set to about 80 percent of the limit, prevents out-of-memory kills. The batch processor at 512 items or 5 seconds keeps export calls efficient. A terminationGracePeriodSeconds longer than the application's own shutdown plus the collector's flush timeout lets the last batch leave. The note says the memory limiter must be set below the container limit, or the kernel kills the collector before the limiter acts. setting starting value why CPU request / limit 50m / 200m batching and export memory request / limit 64Mi / 128Mi room for a queue memory_limiter ~80% of the limit refuse before OOM batch 512 items or 5 s efficient exports terminationGracePeriod app + flush + margin last batch leaves set the memory limiter below the container limit, or the kernel acts first
A sidecar is small, but its limits must leave room for the memory limiter to act before the kernel does.

Configuration options

Setting Value Why
restartPolicy on the init container Always makes it a native sidecar, with ordering
Application endpoint http://127.0.0.1:4317 never leaves the pod
Sidecar memory limit 256 MiB queue plus working space
memory_limiter.limit_mib 192 refuses before the container limit
terminationGracePeriodSeconds 45 drain, then flush, then sidecar drain
Sidecar CPU request 50m protobuf work is bursty and small
Exporter target the gateway credentials stay in one place

Verification

The property worth verifying is the shutdown window, because it is the one that silently does not work.

# delete a pod and watch the ordering in both containers' logs
kubectl delete pod checkout-7d9f8c5b6-xk2lm --grace-period=45 &
kubectl logs -f checkout-7d9f8c5b6-xk2lm -c app --tail=5
kubectl logs -f checkout-7d9f8c5b6-xk2lm -c otel-sidecar --tail=5

Expected Output: the application's flush is acknowledged by a collector that is still running.

app       telemetry flushed: 318 spans
sidecar   otelcol_exporter_sent_spans increased by 318
sidecar   everything flushed, exiting

If the sidecar's log ends before the application's flush line, the ordering is not in effect — usually because the collector is an ordinary container rather than an init container with an always restart policy.

Common mistakes

The collector exits before the application flushes. Error signature: export failures in the final seconds of every pod termination. Root cause: an ordinary container, so termination order is unspecified. Remediation: declare it as a native sidecar, as in step 1.

The pod is evicted for memory and the application is blamed. Error signature: an out-of-memory kill whose container is the collector. Root cause: no memory limiter, so the queue grew into the pod's budget. Remediation: set the limiter below the container limit and alert on refused records.

Every service gets a sidecar by default. Error signature: several gigabytes per node in idle collector processes. Root cause: adopting the pattern fleet-wide rather than selectively. Remediation: keep the node agent as the default and reserve sidecars for services that can name the reason.

The sidecar's own metrics are never scraped. Error signature: nobody notices that one deployment's collector has been refusing records for a week. Root cause: the sidecar exposes its telemetry on a pod-local port that the cluster's scrape configuration does not cover, because that configuration was written for the node agent. Remediation: add the sidecar's port to the scrape annotations on the pod, and treat its refused and dropped counters exactly as you treat the node agent's.

The application keeps serving after the flush. Error signature: spans produced after the provider was shut down, silently discarded. Root cause: the termination handler flushes telemetry before the server stops accepting requests. Remediation: drain the server first and flush last, so nothing is produced after the provider closes.

Frequently Asked Questions

Does a sidecar collector cost a lot of memory?

Its floor is meaningful: 50 to 100 MiB per pod even when idle, plus whatever queue you configure. Across a deployment of forty replicas that is a few gigabytes doing nothing most of the time, which is the main argument for a shared per-node collector instead.

Why a native sidecar rather than an ordinary container?

Ordering. A native sidecar — an init container with restartPolicy Always — starts before the application container and is terminated after it, so the application can export during its own shutdown. An ordinary container has no ordering guarantee, and the collector is frequently killed first.

Should the sidecar export to a gateway or straight to the backend?

To a gateway if one exists, for the same reason the node agent does: credentials and routing belong in one place. Straight to the backend is defensible only when the sidecar is the only collector tier you run.

Can a sidecar and a node agent coexist?

Yes, and it is a reasonable arrangement: most workloads use the node agent, and the two or three services that need isolation get a sidecar that forwards to the same gateway. The routing difference is one environment variable.

Related Guides