Python Telemetry Pipelines and Delivery: Architecture and Operations Guide

Instrumenting a Python service produces telemetry. Getting that telemetry to a place where somebody can query it during an incident is a separate engineering problem, and it is the one that fails quietly. This guide is for backend engineers and SREs who have working instrumentation and now need the delivery path to hold up: it covers shipping logs off the process, where to run collectors and in what topology, what actually happens under backpressure and how much you can promise about delivery, controlling volume and cost, and the awkward cases of serverless and batch work. It assumes the instrumentation itself is in place — if it is not, start with distributed tracing and OpenTelemetry in Python, Python logging fundamentals and Python metrics and instrumentation.

Key architectural decisions

• Where the process hands off. Standard output collected by an agent, or a direct network export. Containers make the first cheap and the second explicit; both are legitimate, and mixing them per signal is normal.
• Agent, gateway, or both. An agent gives every process a local endpoint that is almost never down. A gateway gives the fleet one place to hold credentials and apply policies that need to see traffic from many processes at once.
• What you promise about delivery. At-most-once is the honest default for telemetry. Anything stronger costs disk, and is worth buying only for the signals that carry business meaning.
• Where volume is cut. In the SDK it saves CPU; in the collector it saves money and can be changed without a deploy. The two cuts are not interchangeable.
• What happens at shutdown. A process that exits without flushing loses the telemetry describing why it exited, which is the telemetry you wanted.

The Shape of a Telemetry Pipeline

Every telemetry pipeline, whatever tools it is built from, is the same four stages: produce, buffer, transport, store. The interesting engineering is entirely in the second and third.

Production happens inside the Python process. A span ends, a log record is emitted, a counter is incremented. At this point the data exists only in memory, and the cost is CPU inside the request path — which is why head sampling and lazy log formatting matter: they are the only controls that make the work not happen.

Buffering is where the synchronous path ends. BatchSpanProcessor puts finished spans in a bounded queue and returns; QueueHandler does the same for log records; the metrics SDK accumulates in place and a reader drains it on an interval. In every case a background worker takes over, and the request thread stops caring. The queue's size is the entire delivery guarantee: when it is full, new data is dropped, and it is dropped silently unless you are watching the counter.

Transport is the network hop, and the one place where a slow backend becomes your problem. An export that takes thirty seconds to time out while the queue fills behind it will drop more data than the outage itself justifies. This is why the exporter's timeout should be shorter than the interval at which the queue would fill — an arithmetic relationship covered in tuning BatchSpanProcessor for throughput and generalised across signals in handling OTLP export retries and timeouts.

Storage is somebody else's problem until it is yours: the backend's ingestion limits are what the retries are retrying against, and its pricing model is what the volume controls are controlling.

Choosing a Collection Topology

There are three arrangements in practice, and the choice is about blast radius rather than throughput.

Direct export sends OTLP from the Python process straight to the vendor's endpoint. It has the fewest moving parts and the worst operational properties: the endpoint's TLS, its authentication header, its rate limits and its outages are all inside your application. Changing backends means a deploy. It is right for a proof of concept and for a single service that nobody is on call for.

Agent runs a collector next to every process — a sidecar container, a DaemonSet pod, or a systemd unit on the host. The application exports to localhost, which cannot be partitioned away from it, and the agent owns everything else. This is the arrangement most fleets should start at, and it is covered in detail in agent versus gateway collector deployment.

Agent plus gateway adds a second tier: a small horizontally scaled pool of collectors that all agents forward to. The gateway is where anything requiring a fleet-wide view belongs — tail sampling, which needs all spans of a trace in one place; routing to several backends; and the credentials you do not want present on every node.

The cost of each tier is one more queue that can fill and one more process to monitor. The benefit is that everything above it stops needing to know about the backend.

Log Shipping: From Process to Store

Logs are the signal with the most delivery options and the fewest guarantees. A Python process can write JSON to standard output and let a collector tail the container's log file; it can write to a file and let a shipper follow it; it can push over the network from a handler; or it can emit through the OpenTelemetry logs pipeline alongside its spans.

For containerised services the default should be standard output, for one reason: it is the only sink that still works when the application is too broken to do anything else. A log line written to stdout survives a process that cannot open a socket, cannot resolve DNS and is out of file descriptors. That is exactly the process whose logs you need. The trade-offs, including what the container runtime does to long lines, are worked through in standard output versus file logging in containers.

The shipper then has one hard job: reassembling multi-line records. A Python traceback is one logical event printed as fifteen physical lines, and a shipper that has not been told so will index fifteen unrelated documents, none of which contains both the exception type and the line that raised it. Handling multi-line tracebacks in log shippers covers the two fixes: teach the shipper the continuation pattern, or — far better — emit the traceback as a JSON string field so it is never multi-line in the first place, which structured logging with the standard library already sets you up to do.

Delivery Guarantees, Retries and Backpressure

The honest description of a default OpenTelemetry pipeline is at-most-once with a best-effort retry. Data is dropped when the queue is full, when the retry budget is exhausted, and when the process dies with data in memory. Every one of those is a deliberate trade, and every one of them is adjustable.

Retries must be bounded and they must back off. An exporter that retries immediately against a struggling backend adds load exactly when load is the problem, and one that retries forever will hold a batch while the queue behind it fills with newer, more relevant data. The OTLP exporters implement exponential backoff with a total deadline; the number worth setting deliberately is that deadline, because it is what decides whether a thirty-second backend hiccup costs you thirty seconds of telemetry or five minutes of it.

Backpressure is what the queue does when the exporter cannot keep up, and Python's SDKs choose to drop rather than block. This is the correct default — the alternative is a telemetry backend outage becoming a request-latency outage — but it means the drop counter is the only evidence. Detecting dropped spans and metrics covers what to scrape and what to alert on.

For the cases where losing data is genuinely unacceptable, the collector can persist its queue to disk, at the cost of an fsync per batch and a volume to manage. That is the right tool for audit-grade records; it is the wrong tool for a debug span.

Shutdown, Flush and Short-Lived Processes

The telemetry that describes a crash, a deploy rollback or an out-of-memory kill is in the queue at the moment the process stops. If nothing flushes it, the record of the most interesting thirty seconds of the day never leaves the host.

A long-lived server needs a termination handler that calls shutdown() on the tracer and meter providers, and a grace period long enough for the final export. A worker under a process manager needs the same, per worker. And a short-lived process — a cron job, a Lambda invocation, a data pipeline task — needs it structurally, because its entire lifetime may be shorter than one export interval. The batch processor's schedule delay is irrelevant to a script that runs for four seconds; what matters is the explicit flush before exit, which is the subject of graceful shutdown and telemetry flush and, for the specifics, logging from cron and batch jobs and metrics from short-lived jobs with the Pushgateway.

Volume, Cardinality and Cost Control

Telemetry cost is three multiplications, and knowing which one you are in tells you which control to reach for.

Traces cost bytes per span multiplied by spans per request multiplied by request rate multiplied by the sample rate. Every term is adjustable, and the sample rate is the one with a linear effect and no other consequence, which is why it is reached for first.

Logs cost bytes per record multiplied by record rate, with no sampling term unless you add one. This is why a single INFO line added to a hot path can change a bill more than an entire tracing rollout, and why rate limiting and sampling noisy loggers belongs in the same conversation as retention.

Metrics cost active series multiplied by samples per series, and are almost entirely independent of traffic. A metrics bill that grows with traffic is a cardinality bug, not a volume problem — see controlling label cardinality in Prometheus.

Cutting in the collector rather than the SDK has one decisive advantage: it is a configuration change, applied fleet-wide, in minutes, with no deploy. Dropping and aggregating metrics in the collector and estimating telemetry volume from a Python service cover the arithmetic and the mechanics.

Production Code Examples

A service configured for a local agent, with bounded queues, a deliberate export deadline, and a flush on termination.

# telemetry.py — one module, imported before anything else creates a span.
import os
import signal
import logging

from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

log = logging.getLogger(__name__)

# 1. The resource is attached once and lands on every signal.
RESOURCE = Resource.create({
    "service.name": os.environ["SERVICE_NAME"],
    "service.version": os.environ.get("SERVICE_VERSION", "unknown"),
    "deployment.environment": os.environ.get("ENVIRONMENT", "dev"),
})

# 2. The endpoint is local. It is an agent, and it is not the backend.
ENDPOINT = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")

_tracer_provider = TracerProvider(resource=RESOURCE)
_tracer_provider.add_span_processor(
    BatchSpanProcessor(
        # 3. A five second deadline: shorter than the time the queue takes to fill.
        OTLPSpanExporter(endpoint=ENDPOINT, timeout=5),
        max_queue_size=4096,          # ~4096 spans of headroom
        max_export_batch_size=512,    # one request per 512 spans
        schedule_delay_millis=5000,   # or every five seconds, whichever first
    )
)
trace.set_tracer_provider(_tracer_provider)

_meter_provider = MeterProvider(
    resource=RESOURCE,
    metric_readers=[
        PeriodicExportingMetricReader(
            OTLPMetricExporter(endpoint=ENDPOINT, timeout=5),
            export_interval_millis=15000,
        )
    ],
)
metrics.set_meter_provider(_meter_provider)


def flush_and_stop(signum=None, frame=None):
    """Drain both queues before the interpreter exits."""
    # 4. shutdown() flushes then stops the worker; it is safe to call twice.
    _tracer_provider.shutdown()
    _meter_provider.shutdown()
    log.info("telemetry flushed", extra={"signal": signum})


signal.signal(signal.SIGTERM, flush_and_stop)

Expected Output: with the agent stopped, the export fails, retries within the deadline, and the queue reports its losses rather than hiding them.

telemetry flushed signal=15
WARNING opentelemetry.sdk.trace.export Exporter failed after 5.0s; 512 spans dropped

The agent configuration the service is exporting into — receiving OTLP, batching, and limiting its own memory before it starts refusing:

# otel-agent.yaml — runs beside the process, owns nothing that changes often.
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  # 1. Refuse new data before the process is killed for memory.
  memory_limiter:
    check_interval: 1s
    limit_mib: 384
    spike_limit_mib: 96
  # 2. Batch so the gateway sees requests, not packets.
  batch:
    timeout: 5s
    send_batch_size: 512
  # 3. Stamp the host once, here, instead of in every application.
  resourcedetection:
    detectors: [env, system]

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_elapsed_time: 120s
    sending_queue:
      enabled: true
      queue_size: 2000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/gateway]

Expected Output: the agent's own metrics, which are what you alert on.

otelcol_receiver_accepted_spans{receiver="otlp"} 184203
otelcol_exporter_sent_spans{exporter="otlp/gateway"} 184203
otelcol_exporter_send_failed_spans{exporter="otlp/gateway"} 0
otelcol_processor_refused_spans{processor="memory_limiter"} 0

Network and Protocol Integration

OTLP is offered over gRPC on port 4317 and over HTTP with a protobuf body on port 4318. The payloads are identical; the difference is entirely in what your network does to them.

gRPC keeps one HTTP/2 connection open and multiplexes every export over it. Per-export overhead is the lowest available, and a fleet of a thousand processes holds a thousand long-lived connections into the agent tier — which is fine against a local agent and is a real consideration against a shared gateway behind a load balancer that rebalances on connection count rather than request count. HTTP/protobuf opens a request per export, which costs more but passes through every proxy, ingress and egress policy that understands ordinary HTTP. When a service exports to localhost, gRPC is the obvious default. When it exports across a boundary somebody else administers, HTTP is the one that will still work next quarter.

Authentication is a header in both cases, and the header is the thing you do not want in the application. An agent-based topology means the application sends nothing but plaintext OTLP to 127.0.0.1, and the agent attaches the credential on the hop that leaves the host. Securing OTLP with TLS and headers works through the certificate and header plumbing on both ends.

Compression is on by default for OTLP over gRPC and worth confirming rather than assuming: span payloads are highly repetitive — the same resource attributes on every record — and gzip typically removes seventy to ninety percent of the bytes. On a link you pay for, that ratio is the difference between the volume estimate and the invoice.

# Two exporters, same payload, different network characteristics.
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter as GrpcSpanExporter,
)
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
    OTLPSpanExporter as HttpSpanExporter,
)

# Local agent: one multiplexed connection, no credential, no TLS.
local = GrpcSpanExporter(endpoint="http://localhost:4317", insecure=True, timeout=5)

# Across an administered boundary: ordinary HTTPS, one request per export.
remote = HttpSpanExporter(
    endpoint="https://otel.example.internal/v1/traces",
    headers={"authorization": f"Bearer {TOKEN}"},
    timeout=10,
)

Expected Output: the exporter's debug logging shows the shape of each hop.

grpc  export ok  batch=512  compressed=41.2 KiB  raw=318.7 KiB  elapsed=38 ms
http  export ok  batch=512  compressed=42.0 KiB  raw=318.7 KiB  elapsed=71 ms

Routing, Tenancy and More Than One Backend

Every fleet eventually needs the same telemetry in two places: a vendor for the on-call view, an object store for long retention, a separate tenant boundary for a regulated workload, or a second vendor during a migration that lasts three quarters. Doing this in application code means every service redeploys whenever the answer changes, and services drift out of step while it happens.

In a collector, it is a routing rule. The same receiver feeds several exporters; a routing processor selects between them on a resource attribute such as deployment.environment or a tenant identifier carried in baggage; and a filter trims the copy that goes to the expensive destination. The application keeps exporting to one endpoint throughout, and knows nothing about any of it. Routing telemetry to multiple backends covers the processor configuration; the point worth carrying here is architectural. The moment two destinations exist, the routing decision belongs outside the release cycle, because it will change more often than the code does.

Tenancy adds one constraint on top: a routing rule that leaks one tenant's records into another tenant's destination is a data incident, not a misconfiguration. The safe pattern is a default route that goes to a quarantine destination rather than to any tenant, so a record with a missing or unrecognised tenant attribute lands somewhere harmless and visible instead of somewhere wrong.

What to Monitor About the Pipeline Itself

A telemetry pipeline fails silently by construction: when it stops working, the symptom is an absence, and an absence looks exactly like a quiet night. Four things are worth a dashboard and an alert.

Queue utilisation in the process. The span processor exposes queue size and a dropped count. A queue that is routinely half full is one traffic spike from dropping, and the fix is either a larger queue or a lower sample rate — knowing which requires the number.

Export failure rate. Failures are normal in small quantities and meaningful in trends. An export failure rate that rises while request latency is flat is the pipeline degrading independently of the service, which is exactly the case you want separated from a real incident.

Collector accepted versus sent. The collector counts what it received and what it forwarded per pipeline. The difference is what it dropped, and it should be zero. A persistent gap with no refused counter usually means a filter is matching more than its author intended.

End-to-end freshness. The most useful pipeline alert is the simplest: the age of the newest record in the backend, per service. It catches every failure mode at once — a dead agent, a full queue, an expired credential, a routing rule sending records to a destination nobody queries — and it is the only check that verifies the whole path rather than one hop of it.

# The four pipeline alerts, in the order they usually fire.
max by (service) (otel_sdk_span_processor_queue_size / otel_sdk_span_processor_queue_capacity) > 0.5

rate(otelcol_exporter_send_failed_spans[5m]) > 0

rate(otelcol_receiver_accepted_spans[5m]) - rate(otelcol_exporter_sent_spans[5m]) > 0

time() - max by (service) (telemetry_last_received_timestamp_seconds) > 300

Expected Output: during a gateway restart, only the second and third fire, and they recover without the fourth ever tripping — which is the signature of a pipeline absorbing a fault rather than passing it on.

PipelineExportFailures   firing   service=checkout  for 40s
PipelineAcceptedVsSent   firing   service=checkout  for 40s
PipelineStale            ok

Common Mistakes

Exporting directly to the vendor from application code. It works until the first credential rotation, the first backend outage that becomes a latency incident, and the first time somebody wants to send a copy somewhere else. Each of those is a deploy instead of a config change.

Leaving the export timeout at its default. A sixty-second timeout on a five-second queue means one slow export discards everything produced during it. The timeout should be short enough that a failure costs one batch.

Treating the collector as stateless plumbing. It has queues, it drops, and it has its own metrics. An unmonitored collector is a place where telemetry disappears with no trace in either system.

Assuming the container runtime preserves long lines. Several runtimes split log lines above a size limit, which turns one JSON object into two invalid ones. Keep records comfortably under the limit, and put large payloads in a span attribute or object storage instead.

Sampling only in the collector. Tail sampling decides what to keep; it does not stop the process producing spans. If the CPU cost of instrumentation is the problem, only head sampling helps.

Forgetting the flush in short-lived processes. A job that exits cleanly in three seconds and never exports is indistinguishable from a job that did no work.

Related Reading

• Log shipping and collection — getting records off the process and into a store intact.
• Collector topology and deployment — agents, gateways and where each one belongs.
• Backpressure, retries and delivery guarantees — what you can honestly promise about telemetry that matters.
• Telemetry cost and data volume control — the three multiplications, and where to cut each one.
• Telemetry from serverless and batch Python — processes too short-lived for the usual machinery.
• Exporters and the OpenTelemetry Collector — the trace-side detail behind the same components.
• Python Profiling and Performance Observability — CPU and memory profiling, contention, and database and I/O performance.

Frequently Asked Questions