Delta vs Cumulative Temporality

Every metric point the OpenTelemetry SDK exports carries a temporality. A cumulative point says "this many events since the process started"; a delta point says "this many since the last export". The two carry the same information, and backends are particular about which they accept — a delta counter sent to a backend expecting cumulative produces rates that are wrong in ways that look plausible. This article explains the difference, what the Python SDK does by default, how to change it, and how to convert between the two in the collector. It belongs to the OpenTelemetry metrics SDK in the Python metrics and instrumentation section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

An OTLP metrics pipeline as in exporting OTLP metrics to the collector.

How each temporality behaves

Cumulative is the Python SDK's default for every instrument. The SDK keeps a running aggregation per attribute combination from the moment the combination is first seen, and each export reports the current totals. Points have a fixed start time — when the aggregation began — and a moving end time. The backend computes rates by subtracting consecutive points, exactly as Prometheus does with scraped counters.

Cumulative export is robust to loss: a dropped export batch means one missing point, and the next point contains everything. It is also what Prometheus-compatible backends expect, including the collector's Prometheus exporter and remote-write exporter. Its cost is memory: the SDK holds every combination it has ever seen for the life of the process, because each must keep accumulating.

Delta reports only what happened since the last export, then resets. Points have a start time equal to the previous export and an end time of now. The backend, rather than subtracting, sums points to produce totals and divides by interval length to produce rates.

Delta export makes ingestion simpler for backends, since each point is self-contained and needs no knowledge of previous points. It frees SDK memory, because combinations that saw no updates in an interval can be dropped. And it is fragile under loss: a batch that fails to export takes its events with it.

Instruments differ in sensible defaults. Up-down counters and observable gauges report current values, and for them cumulative is the only meaningful temporality — a delta of a queue length is rarely what anyone wants. The SDK's delta preference therefore applies delta to counters and histograms and keeps cumulative for up-down counters, which matches what delta-preferring backends expect.

Implementation steps

Step 1 — Find what the backend expects. Prometheus, Mimir, Thanos, Cortex and most Prometheus-compatible stores want cumulative. Several commercial backends prefer delta for counters and histograms. The backend's OTLP ingestion documentation states which; when it accepts both, cumulative is the safer default because of its loss tolerance.

Step 2 — Configure the SDK. The environment variable sets the preference for the OTLP exporter without code changes.

export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta        # or cumulative, lowmemory

Or in code, per instrument type:

from opentelemetry.sdk.metrics import Counter, Histogram, UpDownCounter, ObservableCounter
from opentelemetry.sdk.metrics.export import AggregationTemporality, PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

exporter = OTLPMetricExporter(preferred_temporality={
    Counter: AggregationTemporality.DELTA,
    Histogram: AggregationTemporality.DELTA,
    ObservableCounter: AggregationTemporality.DELTA,
    UpDownCounter: AggregationTemporality.CUMULATIVE,
})
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=60_000)

The lowmemory preference is a middle ground: delta for synchronous counters and histograms, where the SDK can drop idle combinations, and cumulative for observable counters, whose values come from outside the SDK and are naturally cumulative.

Step 3 — Convert in the collector when sources and backend disagree. A fleet may have services exporting cumulative, a backend wanting delta, or the reverse. The collector converts:

processors:
  cumulativetodelta: {}                # cumulative sources → delta backend
  deltatocumulative:                   # delta sources → Prometheus-style backend
    max_stale: 5m
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [deltatocumulative, batch]
      exporters: [prometheusremotewrite]

Both processors hold state per series in the collector — deltatocumulative the running total, cumulativetodelta the previous value — so every point for a series must reach the same collector instance. Behind a load-balanced gateway, that means routing by series, as in routing telemetry to multiple backends, or doing the conversion in an agent collector running beside each service.

Restarts, resets and start times

A Python process that restarts loses its cumulative state, and its counters begin again at zero. The start time on each point changes to the new process start, which tells the backend a reset happened. Prometheus-compatible backends also detect resets when a counter value decreases, and compute rate and increase correctly across them — the few seconds between the last point before the restart and the first after are lost, which is usually negligible.

Delta export has no reset problem, since each point stands alone. It has a different edge case: at shutdown, the events recorded since the last export exist only in memory. Without a final flush they are lost, which is why MeterProvider.shutdown() in a shutdown hook matters more for delta pipelines — graceful shutdown and telemetry flush covers the sequence.

Multi-process servers interact with both. Each Gunicorn worker is its own process with its own start time, and recycling a worker is a restart for that worker's series. With cumulative export and a worker identity in the resource, each recycled worker produces a new series, and old ones become stale. With delta export and no worker identity, all workers' points for a series arrive as separate deltas that the backend must sum — which delta backends do correctly, and which makes per-worker identity unnecessary for them.

Choosing, in one decision

Most teams do not need to think about temporality beyond one choice made once for the fleet, and the choice follows from the backend and the pipeline shape.

The rule that avoids most trouble is uniformity. A fleet where some services export delta and others cumulative into the same pipeline needs conversion rules that match each source, and a new service configured the wrong way silently produces wrong numbers until someone notices a dashboard. Setting the preference through a shared base image or a common telemetry package, rather than per service, keeps the fleet consistent.

Histograms deserve a specific check. Exponential histograms and explicit-bucket histograms both support both temporalities, and conversion processors handle them, but some backends accept one histogram type only in one temporality. Sending a test histogram through the full pipeline and inspecting the stored buckets is quicker than reading every component's documentation.

Configuration options

Verification

Record a known number of events, then check the backend:

counter = meter.create_counter("verification.events")
for _ in range(600):
    counter.add(1)

In a Prometheus-compatible backend, increase(verification_events_total[5m]) should return 600 after the next export. Restart the process, record 600 more, and the increase over a window covering both should be 1 200 — not 600, which would mean the reset was misread, and not a huge number, which would mean delta points were interpreted as cumulative.

Expected Output:

increase(verification_events_total[10m]) => 1200

A delta exporter pointed at a Prometheus backend without conversion typically shows a sawtooth: each point is read as a small cumulative total, and rate sees constant resets. That pattern is the signature of a temporality mismatch.

Common mistakes

Delta into a cumulative backend. Error signature: counters that sawtooth and rates far below reality. Root cause: each delta read as a total. Remediation: cumulative in the SDK, or deltatocumulative in the collector.

Cumulative into a delta backend. Error signature: totals that grow quadratically. Root cause: each running total summed as if it were an increment. Remediation: the delta preference, or cumulativetodelta.

Conversion behind a load balancer. Error signature: converted series with gaps and jumps. Root cause: points for one series split across collector instances. Remediation: route by series or convert at the agent.

No flush at shutdown with delta. Error signature: the last interval's events missing after every deploy. Root cause: unexported deltas lost with the process. Remediation: MeterProvider.shutdown() on exit.

Cumulative with rotating attributes. Error signature: SDK memory growing through the day. Root cause: aggregations kept for every combination ever seen. Remediation: bounded attributes, or lowmemory.

Mixed temporality across the fleet. Error signature: one service's rates correct and another's wildly off in the same pipeline. Root cause: services configured individually and inconsistently. Remediation: set the preference in a shared base image or telemetry package.

Assuming histograms convert like counters. Error signature: histograms missing or malformed at the backend while counters are fine. Root cause: a backend accepting a histogram type only in one temporality. Remediation: send a test histogram end to end and inspect the stored buckets.

Frequently Asked Questions