Python Observability

Detecting Dropped Spans and Metrics

The failure mode of a telemetry pipeline is silence, and silence is indistinguishable from a quiet service. Everything in this page exists to convert that silence into a number somebody can alert on. It covers the counters each tier already produces, the reconciliation that localises a loss, and the synthetic probe that measures delivery independently of traffic. It is a task article under backpressure, retries and delivery guarantees, part of the Python telemetry pipelines and delivery section.

Four numbers that should agree Four counters are drawn along the pipeline. The first is spans created, known to the application's SDK. The second is spans accepted, known to the agent's receiver. The third is spans sent, known to the gateway's exporter. The fourth is spans stored, known to the backend. In a healthy pipeline all four are equal. When they are not, the pair that disagrees names the tier responsible: created above accepted means the application dropped, usually from a full queue; accepted above sent means a collector dropped, usually from a memory limiter or a retry deadline; sent above stored means the backend rejected, usually a quota or a malformed payload. A fifth arrow marks the case no counter covers, which is data that was never produced because the process died before creating it or the instrumentation was removed, and which is only visible as unexpected quiet. the four counts, and what each gap means created SDK accepted agent receiver sent gateway exporter stored backend created > accepted the application's queue filled accepted > sent a collector refused or gave up sent > stored the backend rejected it the case no counter covers nothing was created — the process died first, the instrumentation was removed, or the agent never started only a staleness check sees this, which is why it is the alert that matters most
Three of the four failures are differences between counters. The fourth is an absence, and it needs a different kind of check entirely.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-prometheus>=0.48b0,<1.0.0"

Implementation

Step 1 — Enable the SDK's internal telemetry. The processors and exporters inside the Python SDK produce metrics about themselves, and in most deployments nobody has turned them on. Without them the application's queue is a black box: you can observe that data is missing but never that the application was the one to discard it, which sends every investigation downstream to the collector, where the evidence is not.

import os
# The SDK's own metrics, exposed on a local port for the platform to scrape.
os.environ.setdefault("OTEL_PYTHON_INTERNAL_METRICS_ENABLED", "true")

from prometheus_client import start_http_server
start_http_server(9464)          # /metrics for the SDK's internal instruments

Step 2 — Scrape the collector's counters from outside its own pipeline. The collector publishes accepted, refused, sent, failed and dropped counters per pipeline and per component. Sending those metrics through the collector being measured is circular: a collector that stops working also stops reporting that it stopped working. Scrape them with the platform's own Prometheus, over a path that does not traverse the telemetry pipeline.

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888      # scraped directly by the platform, not via OTLP

Step 3 — Reconcile the counts across tiers. Individually each counter says whether one component is healthy. Compared, they say which component lost data, which is a much more useful statement during an incident. The comparison is a subtraction and it only works if the counters are scraped with the same resolution and labelled consistently by service.

# spans the application created but no agent ever accepted
sum by (service) (rate(otel_sdk_span_processor_processed_spans_total[5m]))
  - on(service)
sum by (service) (rate(otelcol_receiver_accepted_spans[5m]))

# spans a collector accepted and never sent onward
sum(rate(otelcol_receiver_accepted_spans[5m]))
  - sum(rate(otelcol_exporter_sent_spans[5m]))

Step 4 — Run a synthetic producer. A tiny job emitting a known number of records per minute through the real pipeline turns delivery into a ratio with a known denominator. This is the only measurement that does not have to be interpreted against what the fleet happened to be doing, and it is the one that detects the failure where production itself stopped.

# probe.py — runs as a CronJob every minute, through the production pipeline.
import os, time
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "telemetry-probe"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(timeout=5)))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)
BATCH = 60                       # a known denominator
for i in range(BATCH):
    with tracer.start_as_current_span("probe") as span:
        span.set_attribute("probe.sequence", i)
        span.set_attribute("probe.minute", int(time.time() // 60))
provider.shutdown()              # flush, because this process is about to exit

Step 5 — Alert on staleness, per service. The most valuable alert in a telemetry pipeline is the simplest: how long since anything arrived from this service. It catches a dead agent, a full queue, an expired credential, a routing rule pointing somewhere nobody looks, and a service that was deployed without instrumentation — all of which produce no counter anywhere and all of which look identical from a dashboard.

# nothing from this service for five minutes
time() - max by (service) (telemetry_last_received_timestamp_seconds) > 300

# the probe delivered fewer than it produced
sum(rate(probe_spans_stored[5m])) / 60 < 0.99

Expected Output: a healthy pipeline reports a full probe, an empty backlog, and no drops anywhere.

probe delivered      60/60
receiver accepted    1_284_200
exporter sent        1_284_200
sdk dropped spans    0
oldest service age   8s
Why a rate alert is not enough Two record-rate charts drawn side by side, both showing a steep fall to near zero at the same moment. The left chart is a genuine drop in traffic overnight, and the pipeline is healthy: the synthetic probe continues to report sixty of sixty delivered every minute throughout, so the absence of application records is real information about the application. The right chart is a broken pipeline, and the application traffic is unchanged: here the probe reports zero of sixty, which distinguishes the two cases immediately. The point made underneath is that no threshold on the application's own record rate can separate these, because the two charts are identical, and that the probe's known denominator is what supplies the missing information. the same chart, two completely different situations quiet night — pipeline healthy probe: 60/60 every minute the absence is real information broken pipeline — traffic unchanged probe: 0/60 — nothing is arriving at all the absence is the pipeline no threshold on the upper chart can tell these apart — the denominator has to come from somewhere else
Two identical charts, two opposite meanings. The probe's known denominator is the only thing that separates them.

Building the alerts that are worth waking for

Not every drop deserves an alert, and a pipeline with an alert per counter trains everyone to ignore all of them. Four alerts cover the meaningful cases, in decreasing order of urgency.

Nothing has arrived from a service in five minutes. This is the one that catches everything, including the failures that increment no counter. It needs a per-service list so that a newly deployed service is noticed rather than silently absent, and it needs the threshold to be generous enough that a slow batch interval does not trip it. In most fleets this single alert has caught more real pipeline failures than every other check combined.

The synthetic probe delivered less than it produced. A ratio below one means loss somewhere in the path, measured independently of traffic. Because the denominator is known, this fires accurately at three in the morning when real traffic is low, which is precisely when rate-based checks are useless.

A tier is dropping. The SDK's dropped count or the collector's refused count above zero, sustained. This is not urgent on its own — a brief spike during a deploy is normal — but a sustained non-zero value means the pipeline is running past its capacity and will lose data during the next incident, when it matters most.

Accepted exceeds sent, persistently. A backlog that does not clear. This one is worth a lower-severity alert rather than a page, because it usually resolves and, when it does not, the staleness alert will follow.

What is deliberately absent from that list is an alert on retries, on export latency, or on queue depth alone. All three fluctuate normally, all three produce noise, and none of them means data was lost. Alerting on them teaches the team that pipeline alerts are ignorable, which is exactly the outcome that makes the important alert useless when it eventually fires.

Loss that no counter will ever report

Three categories of loss are invisible to every counter in the pipeline, and they account for a surprising share of the cases where an engineer is certain a record existed and cannot find it.

Data that was never produced. A sampler that discarded the trace at its root, a log statement below the configured level, an instrumentation library that was not installed in this service. Nothing dropped anything; the record simply never existed. This is by far the most common answer to "where did my span go", and the way to rule it in or out quickly is to check the sampling decision and the effective log level for that service before looking at any pipeline component at all.

Data lost with the process. A pod terminated with records in its queue, as covered in graceful shutdown and telemetry flush. The counters that would have reported the loss died with the process that held them, so the evidence is the absence of a flush line in the terminating pod's final log — which is why that line is worth emitting explicitly.

Data delivered somewhere unexpected. A routing rule that matched differently than intended, a tenant attribute that was missing, an index pattern that changed. Every counter reports success because delivery genuinely succeeded; the records are in a place nobody is querying. The reconciliation in step 3 does not catch this either, since sent and stored both increment. Only the per-destination counts described in routing telemetry to multiple backends make it visible, and only if somebody is looking at the destination that should be empty.

The counter at each stage A table of pipeline stages and the counter that reports drops at each. In the Python SDK, the batch span processor drops spans when its queue is full, reported through the SDK's own logs and, in recent versions, a dropped-spans metric. At the collector receiver, refused spans are counted by otelcol_receiver_refused_spans. At the collector processor, the memory limiter's refusals appear in otelcol_processor_refused_spans. At the exporter, failed sends after retries appear in otelcol_exporter_send_failed_spans, and queue overflow in otelcol_exporter_enqueue_failed_spans. The note says an alert on each non-zero rate turns silent loss into a signal. stage drop counter SDK batch processor SDK warning logs · dropped-spans metric collector receiver otelcol_receiver_refused_spans memory limiter otelcol_processor_refused_spans exporter, after retries otelcol_exporter_send_failed_spans exporter queue full otelcol_exporter_enqueue_failed_spans an alert on each non-zero rate turns silent loss into a signal
Every stage that can drop telemetry reports it somewhere. The work is collecting those counters and alerting on them.

Configuration options

Signal Source Threshold Severity
Records arrived, per service backend none for 5 min page
Probe delivery ratio synthetic producer below 0.99 page
SDK dropped spans application /metrics above 0, 10 min ticket
Collector refused collector :8888 above 0, 10 min ticket
Accepted minus sent collector :8888 positive, 15 min ticket
Queue depth collector :8888 above 50% capacity dashboard only
Retries collector :8888 never alert dashboard only

Verification

The check is that the detection works, not that the pipeline does. Break it deliberately and confirm each alert fires.

# 1. stop the agent on one node and confirm the staleness alert fires
kubectl delete pod -l app=otel-agent --field-selector spec.nodeName=node-12

# 2. confirm the probe notices before anybody does
kubectl logs job/telemetry-probe --tail=2

Expected Output: the probe reports a shortfall within one cycle, well before a human would notice missing traces.

probe delivered 0/60 for minute 29348051
TelemetryStale  firing  service=checkout  node=node-12  for 5m

Common mistakes

Scraping the collector's metrics through the collector. Error signature: a collector outage with no alert, because the alerting data travelled the broken path. Root cause: circular monitoring. Remediation: scrape the collector's own endpoint directly from the platform's metrics stack.

No SDK internal metrics. Error signature: investigations that always end at the collector because the application's behaviour is unknown. Root cause: the SDK's self-telemetry is off. Remediation: enable it and expose it alongside the application's own metrics.

Alerting on rate without a denominator. Error signature: pages every night at low traffic, and silence during a real failure at midday. Root cause: a threshold on absolute record rate. Remediation: compare against the service's own recent history, and use the probe for the absolute check.

Counters that cannot be compared. Error signature: a reconciliation query that returns nonsense. Root cause: the tiers label services differently, so the join produces no matches. Remediation: standardise the service label at the source, in the resource attributes, so every tier reports the same name.

One alert per counter. Error signature: a channel nobody reads. Root cause: alerting on everything measurable rather than on everything meaningful. Remediation: keep the four alerts above and leave the rest on a dashboard.

Frequently Asked Questions

How do I know whether my application is dropping spans?

The span processor exposes a queue size and a dropped count through the SDK's internal metrics, which are off by default in most setups. Once enabled, a non-zero dropped counter is unambiguous evidence, and a queue that sits above half full is the warning that precedes it.

Why is a missing-data alert better than a failure alert?

Because it catches the failures nothing counts. An agent that never started, a routing rule sending records to a destination nobody queries, a credential that expired — none of these increment a drop counter anywhere, and all of them produce the same symptom, which is that nothing has arrived recently.

Where should the collector's own metrics go?

To a destination that does not depend on the collector being healthy. Scraping them through the same pipeline they describe means a collector failure removes the evidence of itself, which is the observability equivalent of a smoke alarm wired to the thing that is on fire.

Does a synthetic producer cost much?

Negligible. A few records a minute through the real pipeline gives a continuous delivery ratio with a known denominator, which is the one measurement that does not have to be interpreted against what the fleet happened to be doing.

Related Guides