Python Observability

Running the OpenTelemetry Collector for Python Services

A Collector is the piece that lets a Python service know nothing about your observability backend. This page covers where to run it, the pipeline that makes it useful, the limits that keep it from becoming the incident, and the metrics that prove it is working. It builds on exporters and the OpenTelemetry Collector, part of the distributed tracing and OpenTelemetry in Python section.

Three places to run a Collector Three deployment shapes compared. As a sidecar, one Collector runs inside each service's pod: the network hop is local to the pod, one service's telemetry burst cannot affect another, and the cost is one Collector process per service replica. As a node agent, one Collector runs per host or Kubernetes node and every service on that node sends to it: the hop stays on the host, the resource cost is per node rather than per replica, and a single noisy service can affect its neighbours on the same node. As a gateway, a separately scaled tier of Collectors receives from the agents: it crosses the network, costs an extra hop, and is the only shape that supports tail sampling, because deciding whether to keep a trace requires every span of that trace to reach the same instance. Most production fleets run agents feeding a gateway, which is why the diagram shows the third as an addition to the second rather than an alternative. where the Collector runs decides what it can do sidecar service collector hop: inside the pod blast radius: one service cost: one process per replica for a service that must not be affected node agent svc a svc b collector hop: on the host blast radius: the node the usual starting point gateway tier agents gateway ×N hop: across the network enables tail sampling a whole trace must reach one instance added to agents, not instead of them the one that decides for most teams tail sampling — keeping every trace that contains an error, discarding the rest — needs a gateway, because the decision cannot be made until every span of the trace has arrived, and spans of one trace come from services on many different nodes
Start with a node agent. Add a gateway when you want tail sampling, which is the one capability the agent tier structurally cannot provide.

Prerequisites

docker pull otel/opentelemetry-collector-contrib:0.108.0

export BACKEND_ENDPOINT=https://ingest.example-backend.com
export BACKEND_TOKEN=…                    # only the Collector ever sees this

The -contrib distribution carries the processors most deployments need — attributes, tail sampling, filters, resource detection. The core distribution is smaller and lacks them.

Implementation

Step 1 — Write the pipeline, in the order the processors must run.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:                       # FIRST — refuse before anything expensive
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100

  resourcedetection:                    # add host, cloud and k8s attributes
    detectors: [env, system]
    timeout: 2s

  attributes/redact:                    # policy, once, for every language
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash
      - key: db.statement
        action: hash

  batch:                                # LAST — everything else has already run
    timeout: 5s
    send_batch_size: 1024
    send_batch_max_size: 2048

exporters:
  otlphttp/backend:
    endpoint: ${env:BACKEND_ENDPOINT}
    headers:
      authorization: "Bearer ${env:BACKEND_TOKEN}"
    sending_queue: { enabled: true, num_consumers: 4, queue_size: 5000 }
    retry_on_failure: { enabled: true, initial_interval: 5s, max_elapsed_time: 300s }

extensions:
  health_check: { endpoint: 0.0.0.0:13133 }

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, attributes/redact, batch]
      exporters: [otlphttp/backend]
  telemetry:
    metrics:
      address: 0.0.0.0:8888             # the Collector's own metrics

memory_limiter first is the load-bearing detail. Under pressure it refuses incoming data with a retryable error, which pushes the problem back to the SDK's bounded queue where it belongs. Placed later, the expensive processors have already run on data the Collector is about to reject.

Step 2 — Redact once, at the Collector. This is the single strongest argument for running one. A db.statement attribute containing bound parameters, an http.request.header.authorization, a user.email — each of those is a policy question, and answering it in the Collector answers it identically for every Python, Go and Java service you run.

  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete                    # gone entirely
      - key: user.email
        action: hash                      # correlatable, not readable
      - key: db.statement
        action: hash

The application-side equivalent — and the reason it is worth doing in both places — is in redacting sensitive data in log records.

Why memory_limiter goes first The same four processors in two orders, under memory pressure. With memory_limiter first, incoming batches are refused immediately with a retryable error before any other processor touches them: the CPU cost of resource detection, attribute hashing and batching is never paid on data that will be rejected, and the refusal propagates back to the sending SDK, which retries or queues. With memory_limiter last, every batch is fully processed — host attributes detected, statements hashed, spans batched — and only then refused, so the Collector spends its scarcest resource under pressure doing work it discards, which deepens the pressure it was trying to relieve. The note added is that refusing is the correct behaviour in both orders; only the cost of refusing differs, and under pressure that cost is the whole problem. the same processors, under memory pressure memory_limiter first memory_limiter refuses here — retryable, and nothing else has run redact batch memory_limiter last resourcedetection hash every statement batch memory_limiter refuses now refusing is correct in both — but in the second, the Collector spent its scarcest resource doing work it then threw away
Both configurations refuse the data. Only one of them refuses it before paying to process it, which under memory pressure is the entire difference.

Step 3 — Give the process real limits. The memory_limiter value must sit below the container limit, or the kernel kills the Collector before the limiter engages.

# kubernetes
resources:
  limits:   { memory: 512Mi, cpu: "1" }
  requests: { memory: 256Mi, cpu: "200m" }

With a 512 MiB container limit, limit_mib: 400 and spike_limit_mib: 100 give the limiter room to act at 300 MiB before the runtime intervenes.

Step 4 — Point the services at it. With a node agent and the Kubernetes downward API:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4317"
  - name: HOST_IP
    valueFrom:
      fieldRef: { fieldPath: status.hostIP }
  - name: OTEL_EXPORTER_OTLP_INSECURE
    value: "true"

The service now holds no backend name, no credential, and no TLS material — which is the outcome the whole exercise is for.

Step 5 — Scrape the Collector's own metrics. The gap between received and sent is the only reliable health signal.

  telemetry:
    metrics:
      address: 0.0.0.0:8888
Reading the accepted-versus-sent gap Three shapes of the gap between spans accepted by the Collector's receiver and spans sent by its exporter. In the healthy shape the two lines are indistinguishable, tracking each other exactly, and the refused counter is flat at zero. In the memory-pressure shape the sent line falls below the accepted line during a load spike while the refused counter climbs by the same amount, which identifies memory_limiter shedding and points at either the limit setting or the volume arriving. In the backend-trouble shape the accepted line is unchanged, the sent line drops, and the refused counter stays at zero — nothing in the Collector rejected anything, so the loss is in the exporter's own queue, meaning the backend is slow or returning errors. Each shape is distinguishable from the other two by the refused counter alone, which is why it is the second thing to look at after the gap itself. accepted vs sent, and what the shape means healthy one line — the two are indistinguishable · refused = 0 memory pressure accepted climbs, sent does not refused climbs by the difference → memory_limiter backend trouble sent drops, refused stays 0 → the exporter queue, not the Collector
The refused counter is what separates the second shape from the third. Without it, both look like "the Collector is losing data".

Configuration options

Setting Where Default Recommended
Deployment shape — — node agent, plus a gateway for tail sampling
memory_limiter position pipeline — first, always
batch position pipeline — last
limit_mib memory_limiter none ~75% of the container limit
sending_queue.queue_size exporter 1000 5000 on a gateway
retry_on_failure.max_elapsed_time exporter 5 min 5 min
health_check extension off on, wired to the readiness probe
Telemetry metrics service.telemetry 8888 scraped and alerted on
Distribution — core -contrib for the useful processors

Verification

curl -s localhost:13133 | jq .          # health
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted|exporter_sent|processor_refused)_spans'

Expected Output:

otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 184203
otelcol_exporter_sent_spans{exporter="otlphttp/backend"} 184203
otelcol_processor_refused_spans{processor="memory_limiter"} 0

Accepted equals sent and nothing is refused — that is a healthy Collector. The two failure shapes to alert on:

otelcol_receiver_accepted_spans{receiver="otlp"} 184203
otelcol_exporter_sent_spans{exporter="otlphttp/backend"} 171904      # a growing gap
otelcol_processor_refused_spans{processor="memory_limiter"} 12299    # and why

A gap with a matching refused count is memory pressure. A gap without one is the exporter's queue, which means the backend is slow or rejecting.

Common mistakes

The Collector is OOM-killed instead of shedding

Error signature: the Collector container restarts under load; memory_limiter metrics show nothing. Root cause: limit_mib was set at or above the container's memory limit, so the kernel acted first. Remediation: set limit_mib to roughly 75% of the container limit and leave headroom for the spike limit.

Tail sampling drops most traces

Error signature: the tail sampler is configured on node agents and keeps a fraction of what its policy should. Root cause: spans of one trace arrive at different agents, so no single instance ever sees a complete trace. Remediation: move tail sampling to a gateway tier with trace-ID-aware load balancing in front of it.

Redaction was configured and the data still arrives

Error signature: the attribute is present in the backend despite an attributes action. Root cause: the processor was defined but not listed in the pipeline's processors array — defining a processor does not enable it. Remediation: check the service.pipelines.traces.processors list, which is the only thing that runs.

Operating it

A Collector is a service you now run, and it deserves the same treatment as any other: a health check, resource limits, a rollout strategy and an owner. Four operational decisions come up in the first month.

Configuration changes. The Collector reloads on restart rather than on file change, so a config update is a rollout. That makes a broken configuration a rolling failure, which is an argument for validating before deploying — the binary has a validate subcommand, and running it in CI on the config file catches the class of error that otherwise takes down telemetry across a node.

Version upgrades. Processor and receiver configuration schemas change between releases more often than most infrastructure, and a component can be renamed or moved between the core and contrib distributions. Pin the version explicitly, read the release notes for the components you use, and upgrade deliberately rather than tracking latest.

Capacity. An agent's cost scales with throughput and with how much work the pipeline does per span. Batching is cheap; attribute processing is moderate; tail sampling is expensive because it holds traces in memory until they complete. A useful habit is to measure the resource cost per thousand spans per second on your own pipeline once, and then size from the span rate rather than from a rule of thumb.

Ownership. The Collector sits between application teams and the observability backend, which makes it exactly the kind of component that ends up owned by nobody. Its configuration encodes policy — redaction, sampling, routing — that application teams depend on and do not control, so an explicit owner and a review path for changes matters more than the technical details.

Decision Default that bites Better
Config changes edit and restart validate in CI, then roll out
Version latest pinned, upgraded deliberately
Sizing a rule of thumb measured per thousand spans/second
Ownership implicit named, with a review path
Failure silent gaps alert on accepted-versus-sent

Redaction is a shared policy

The single most valuable thing a Collector does for a mixed-language fleet is enforce one data policy in one place. That is worth designing rather than accumulating: a list of attribute keys that are always deleted, a list that is always hashed, and a documented reason for each.

processors:
  attributes/policy:
    actions:
      - key: http.request.header.authorization
        action: delete                      # credentials, never stored
      - key: http.request.header.cookie
        action: delete
      - key: user.email
        action: hash                        # correlatable, not readable
      - key: db.statement
        action: hash                        # may contain literal values

Two caveats worth knowing. The policy applies to what reaches the Collector, so anything the application chose not to send is not covered by it and anything the application sent has already left the process — which is why in-process redaction remains worth doing for the highest-sensitivity fields. And an attribute deleted here is gone for everyone, including the team that was using it legitimately, so the list deserves the same review as any other shared configuration.

Testing the pipeline

A Collector configuration is worth testing the way any other configuration is: send known input, assert on known output. The debug exporter makes that straightforward — point a test pipeline at it, send a span carrying every attribute the policy touches, and assert that the rendered output contains what it should and omits what it should not. That test catches a processor defined and not listed, an action with a typo'd key, and a pipeline whose processor order changed — all three of which fail silently in production and all three of which are cheap to catch here.

Frequently Asked Questions

Sidecar, node agent, or gateway?

A node agent — one Collector per host or per Kubernetes node — is the usual starting point: one hop for every service on the node, one config to manage, and per-node resource cost. A sidecar gives per-service isolation and is worth it when one noisy service must not affect its neighbours. A gateway is a separate horizontally-scaled tier and is what you add when you need tail sampling, which requires all spans of a trace to arrive at the same instance. Larger deployments run agents feeding a gateway.

Why does processor order matter in the pipeline?

Because the processors run in the order listed, and each sees what the previous one produced. memory_limiter belongs first so it can refuse data before anything expensive touches it. batch belongs last so everything that modifies spans has already happened and batching is not undone. Putting batch before redaction still works but does the redaction on larger payloads for no benefit.

Does the Collector persist data if the backend is down?

Not by default — an exporter's queue is in memory and is lost when the process restarts. The file storage extension adds a persistent queue for exporters that support it, which converts a backend outage from data loss into delayed delivery. It costs disk and adds a failure mode of its own, so it is worth enabling for a gateway tier and rarely worth it for an agent.

How do I know the Collector is healthy?

Scrape its internal metrics and compare accepted against sent for each signal. A growing gap between otelcol_receiver_accepted_spans and otelcol_exporter_sent_spans means something in the middle is dropping — usually memory_limiter shedding under pressure or an exporter queue that is full. Also watch otelcol_processor_refused_spans, which is memory_limiter saying no explicitly.

Related Guides