Collector Topology and Deployment for Python Telemetry

A collector is not a requirement for telemetry to work, which is why it is so often added late and so rarely regretted. This guide covers what each deployment tier actually buys a fleet of Python services, what it costs to run, and how to size and scale it. It is part of the Python telemetry pipelines and delivery section, and it goes deeper than the trace-specific treatment in running the OpenTelemetry Collector for Python services by covering all three signals and the arrangements that serve a whole fleet rather than one service. The focused articles in this topic are Agent vs Gateway Collector Deployment, Routing Telemetry to Multiple Backends, Running the Collector as a Kubernetes Sidecar and Securing OTLP with TLS and Headers.

Prerequisites

The application needs only the OTLP exporters; everything else is deployment configuration.

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

# the only endpoint the application ever knows
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="checkout"

Concept and architecture

A collector is three things bolted together: receivers that accept data, processors that transform it, and exporters that send it on, wired into per-signal pipelines. Everything about topology follows from where you place that assembly relative to the processes producing data.

An agent is a collector with a local receiver. Its defining property is not what it does but where it is: on the same node, or in the same pod, as the process exporting to it. That means the network hop between application and collector cannot be partitioned, cannot be slow for reasons unrelated to the node, and does not need credentials. The application's exporter therefore almost never fails, which in turn means its queue almost never fills, which is the whole reason the arrangement reduces data loss.

A gateway is a collector with a fleet-wide view. Its defining property is that data from many processes converges on it. That view is required for exactly three things: tail sampling, which cannot decide whether to keep a trace until it has seen all of it; routing rules that must be identical everywhere; and holding credentials in a small number of places. If a fleet needs none of those, a gateway is a component to operate for no benefit.

The processors are where the value is. A memory limiter that refuses data before the process is killed, a batch processor that turns many small exports into few large ones, a resource detector that stamps host and cloud attributes once, and a filter that drops what nobody queries. Each of these could be done in the application; none of them should be, because each would then require a deploy to change.

Step-by-step implementation

Step 1 — Deploy the agent as a DaemonSet. One collector per node, reachable from every pod on that node through the node's address. The cost is one pod per node; the benefit is that no application ever talks to anything remote.

# agent DaemonSet configuration — the parts that matter
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 384          # refuse here …
    spike_limit_mib: 96     # … 96 MiB before the 512 MiB pod limit
  resourcedetection:
    detectors: [env, system, eks]
    timeout: 2s
  batch:
    timeout: 5s
    send_batch_size: 512
    send_batch_max_size: 1024

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    sending_queue: { enabled: true, queue_size: 2000 }
    retry_on_failure: { enabled: true, max_elapsed_time: 120s }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp/gateway] }
    metrics: { receivers: [otlp], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp/gateway] }
    logs:    { receivers: [otlp], processors: [memory_limiter, resourcedetection, batch], exporters: [otlp/gateway] }

Step 2 — Point the application at the node, not at a service. A Kubernetes service address load-balances across nodes, which defeats the purpose: the export leaves the node and can be partitioned. The node's own address, injected from the downward API, keeps it local.

env:
  - name: NODE_IP
    valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(NODE_IP):4317"

Step 3 — Add the gateway when something needs the fleet-wide view. The gateway's configuration differs from the agent's in exactly the places the view matters: it holds credentials, it can hold a tail sampling processor, and it fans out to more than one destination.

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  otlp/vendor:
    endpoint: ingest.vendor.example:443
    headers: { authorization: "Bearer ${env:VENDOR_TOKEN}" }
  otlp/archive:
    endpoint: archive-collector.observability.svc:4317

Step 4 — Size the memory limiter against the pod limit, not against traffic. The limiter exists to make the collector refuse data rather than be killed holding it. Set the limit below the container's memory limit with a spike allowance in between, so the collector's own backpressure engages before the platform's does.

Step 5 — Scale the gateway on queue depth. CPU is a poor autoscaling signal for a component whose failure mode is waiting. The exporter's queue size, as a fraction of its capacity, rises exactly when more replicas would help and stays flat when they would not.

metrics:
  - type: Pods
    pods:
      metric: { name: otelcol_exporter_queue_size }
      target: { type: AverageValue, averageValue: "400" }

Configuration reference

Async and concurrency considerations

From the Python process's point of view the collector's topology is invisible except through one number: how long an export takes. That number matters because the BatchSpanProcessor exports from a single background thread, so a slow export does not merely delay one batch — it stops the queue draining at all while it is in progress.

Against a local agent, an export completes in single-digit milliseconds and this is a non-issue. Against a remote endpoint across an internet path, an export can take hundreds of milliseconds, and a timeout under failure can take the whole configured timeout. That is the concrete mechanism by which direct export loses data: the queue fills during a slow export because nothing is draining it.

The same reasoning applies to the metrics reader, which exports on an interval from its own thread, and to the log record processor. All three benefit from the same property, which is that the remote hop happens in a different process.

One consequence is worth stating for asyncio services specifically: none of these exports happen on the event loop. The SDK's processors use dedicated threads, so a slow collector cannot cause event loop lag. What it can do is consume the GIL while serialising protobuf, which shows up as a small, steady CPU cost rather than a latency spike.

Production code examples

The application side, complete, with the endpoint supplied entirely by the environment so the same image runs in every topology:

# telemetry.py — identical in dev, staging and production.
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# 1. No endpoint literal anywhere: OTEL_EXPORTER_OTLP_ENDPOINT drives it.
provider = TracerProvider(resource=Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
    "deployment.environment": os.environ.get("ENVIRONMENT", "dev"),
}))

# 2. Timeout short because the hop is local; if it is not, the topology is wrong.
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(timeout=5),
    max_queue_size=4096,
    max_export_batch_size=512,
))
trace.set_tracer_provider(provider)

Expected Output: the agent's receiver counter climbing in step with the application's spans, with no failures.

otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 98422
otelcol_exporter_sent_spans{exporter="otlp/gateway"} 98422
otelcol_exporter_queue_size{exporter="otlp/gateway"} 12

A health check that distinguishes "the collector is running" from "the collector is delivering", which are not the same question:

# liveness: the process answers
curl -sf localhost:13133/ >/dev/null && echo "up"

# usefulness: it is sending as fast as it is receiving
curl -s localhost:8888/metrics | awk '
  /^otelcol_receiver_accepted_spans/ { in_ = $2 }
  /^otelcol_exporter_sent_spans/     { out = $2 }
  END { printf "accepted %d sent %d backlog %d\n", in_, out, in_ - out }'

Expected Output:

up
accepted 98422 sent 98422 backlog 0

A backlog that grows monotonically is the earliest available signal that a tier is not keeping up, and it appears well before anything is dropped.

Sidecar or DaemonSet

Once an agent tier is decided on, the next question is whether it runs once per node or once per pod, and the honest answer for most fleets is once per node.

A DaemonSet amortises the collector's fixed cost — its process, its memory floor, its configuration reload machinery — across every pod on the node. On a node running thirty pods that is thirty times cheaper than a sidecar arrangement, and the configuration exists in one place per cluster rather than in every deployment manifest. The trade is that the node's collector is shared: one service producing an extreme volume of telemetry consumes buffer that its neighbours need, and a collector restart affects every pod on the node at once.

A sidecar reverses both properties. The collector's lifecycle is tied to the pod, so a restart affects one service; the buffer is dedicated, so a noisy service starves only itself; and the configuration can differ per service without a selector. It costs a container per pod — typically 50 to 100 MiB of memory floor even when idle — and it multiplies the number of collector instances to upgrade.

Three situations genuinely justify a sidecar. A service whose telemetry volume is an order of magnitude above its neighbours, where sharing is unfair rather than merely shared. A tenancy boundary where one team's telemetry must not transit a process another team can configure. And a service that needs a processor nobody else does — a bespoke filter, a redaction rule tied to that application's data — where putting it in the shared agent would impose it on everyone.

Everything else is better served by a DaemonSet plus per-service limits at the source, which is a smaller lever applied where the problem actually is. If one service is drowning the node's collector, the fix that generalises is rate limiting at the application or a lower sample rate, not a dedicated collector to absorb output nobody reads.

Operating the tier

A collector is a service, and it needs the same treatment as any other service you are on call for. Four operational properties do most of the work.

Its own telemetry is not optional. A collector exports metrics about itself: accepted, refused, sent, failed and dropped, per pipeline and per component. These are the only numbers that distinguish a healthy quiet night from a pipeline that stopped working, and they should be scraped by something that is not itself downstream of the collector — otherwise a collector failure takes its own alerting with it.

Configuration changes are deploys. A collector reads its configuration at start, and most deployments reload by restarting. That restart discards whatever is in memory, so a routine configuration change loses a few seconds of telemetry unless the queue is persistent. This is fine and worth knowing, because it explains the small gaps that appear after every platform change and are otherwise investigated as incidents.

Version skew matters less than it looks. Collector components move quickly and configuration keys do get renamed, but the wire protocol is stable: an old agent forwarding to a new gateway, or the reverse, works. Upgrade the tiers independently and do not treat them as a coupled release.

Capacity is memory, not CPU. A collector's CPU cost is dominated by deserialising and reserialising protobuf, which is roughly linear in bytes and rarely the constraint. Memory is the constraint, because it holds the queues, and the failure when it runs out is abrupt. Sizing therefore starts from the queue arithmetic — export rate multiplied by the outage duration to absorb — rather than from a CPU target.

Getting there from direct export

Most fleets arrive at a topology rather than choosing one, and the migration from direct export has a shape worth following because it can be done without a synchronised change.

The first step is to deploy the agent tier while every application still exports directly. Nothing routes through it, so nothing can break, and it gives the platform team a chance to get the DaemonSet, its resource limits and its own monitoring right against real conditions.

The second step is to change the endpoint environment variable, service by service, at whatever pace the owning teams are comfortable with. Because the endpoint is an environment variable rather than code, this is a deployment configuration change and reversible in seconds. A service that moves and misbehaves moves back without a code change, which is what makes the migration safe enough to do during business hours.

The third step is to remove the backend credentials from application configuration, which can only happen once the last service has moved. This is the step that delivers most of the security benefit and the one most often forgotten, leaving unused tokens in a dozen secret stores.

The fourth step, if it is needed at all, is the gateway. It is worth deferring until something concrete requires it — a second backend, a tail sampling policy, or a credential rotation that is painful because it touches too many places. Adding a tier in anticipation of a requirement means operating it for months before it earns anything, and the agent tier already delivers the loss reduction that motivated the work.

Common mistakes

Pointing the application at a cluster service instead of the node. Error signature: export latency that varies with cluster load, and occasional failures during node maintenance. Root cause: the "local" hop is not local. Remediation: use the host IP from the downward API, as in step 2.

No memory limiter. Error signature: the collector container killed for exceeding its memory limit, repeatedly, under load. Root cause: the queue grew until the platform intervened. Remediation: set the limiter below the container limit so the collector refuses first, and treat refusals as the signal to scale.

Unbounded retry against a failing backend. Error signature: the collector's queue full, its exporter busy, and nothing progressing. Root cause: max_elapsed_time unset or very high, so one batch occupies the exporter indefinitely. Remediation: bound it; a batch abandoned after two minutes costs far less than a queue that never drains.

Autoscaling the gateway on CPU. Error signature: replicas stable while the queue is full. Root cause: a collector waiting on a slow backend uses very little CPU. Remediation: scale on queue size, which is the number that actually moves.

Running one collector for the whole cluster. Error signature: a single deployment with two replicas serving every pod, and export latency that rises with cluster size. Root cause: the agent tier was skipped and the gateway was asked to do both jobs, so every application's hop crosses the network after all. Remediation: keep the shared tier, add the per-node tier beneath it, and let each do the job its position makes it good at.

Tail sampling on the agent. Error signature: traces missing most of their spans, or sampling decisions that disagree between services. Root cause: an agent sees only the spans produced on its node, and a trace spans many nodes. Remediation: tail sampling belongs on the gateway, behind a load balancer that routes by trace identifier so a whole trace reaches one replica.

Frequently Asked Questions