Recording Counters and Histograms with OpenTelemetry

The day-to-day work of metrics instrumentation is deciding which instrument fits a measurement and calling it with the right attributes: a Counter for events you tally, a Histogram for values whose distribution matters, and an observable gauge for state you sample. This walkthrough is for backend engineers and SREs who already have a metrics pipeline running and now need the instrument calls in application code to be correct, cheap, and stable over time. It sits under the OpenTelemetry metrics SDK, which sets up the MeterProvider this page assumes is already in place, and is part of the broader Python metrics and instrumentation reference. Once instruments are recording, exporting OTLP metrics to the Collector covers the hop that ships their aggregated values.

Prerequisites

You need a configured MeterProvider with at least one metric reader attached; the OpenTelemetry metrics SDK guide covers the full bootstrap. Install the SDK with a pinned range so the metrics data model and the view API stay stable across upgrades.

pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"

export OTEL_SERVICE_NAME="checkout-api"          # identity stamped on every series
export OTEL_METRIC_EXPORT_INTERVAL="15000"       # how often instruments are collected, ms
export OTEL_EXPORTER_OTLP_ENDPOINT="otel-collector:4317"

If you are still deciding whether OpenTelemetry instruments or a prometheus_client registry belong in this service, OpenTelemetry vs Prometheus for Python metrics frames that choice before you write any instrument code.

Implementation

Step 1 — Get a Meter and create instruments once. The Meter mints every instrument and stamps the instrumentation scope (name and version) on exported data. Create the meter and its instruments at module scope, not inside a request handler: instrument creation is a registration, and re-registering the same name on every request logs duplicate-instrument warnings and wastes work on the hot path.

from opentelemetry import metrics

meter = metrics.get_meter(__name__, "1.0.0")

Step 2 — Create and increment a Counter. A Counter is monotonic; pass only non-negative deltas. Attributes split the total into series, so keep them bounded.

request_counter = meter.create_counter(
    "http.server.request.count",
    unit="{request}",
    description="Total HTTP requests handled",
)

# On each handled request:
request_counter.add(1, {"http.route": "/checkout", "http.status_code": 200})

Names follow the semantic conventions — dotted, lowercase, no units in the name — and the unit field uses UCUM, where curly-brace annotations like {request} mark a dimensionless count. Getting these right at creation time matters more than it looks: renaming an instrument later breaks every dashboard and alert built on it, because the backend treats the new name as an entirely new series.

For values that rise and fall, use an UpDownCounter, which accepts negatives:

inflight = meter.create_up_down_counter("http.server.active_requests", unit="{request}")
inflight.add(1, {"http.route": "/checkout"})   # request started
inflight.add(-1, {"http.route": "/checkout"})  # request finished

Pair those two calls in a try/finally block, or the counter drifts upward every time a handler raises and the decrement is skipped.

Step 3 — Create and record a Histogram. A Histogram captures the distribution of a value, most often latency. Record the observed value once per event, timing it with time.perf_counter() — a monotonic clock that is immune to wall-clock adjustments mid-request.

import time
from contextlib import contextmanager

latency = meter.create_histogram(
    "http.server.duration",
    unit="ms",
    description="HTTP server request duration",
)

@contextmanager
def observe_duration(histogram, attributes):
    start = time.perf_counter()
    try:
        yield
    finally:
        # Record in finally so failed requests appear in the distribution too.
        histogram.record((time.perf_counter() - start) * 1000.0, attributes)

with observe_duration(latency, {"http.route": "/checkout", "http.status_code": 200}):
    handle_checkout()

Recording in the finally block is deliberate: excluding failed requests makes the distribution look healthier than the service actually is, because timeouts and 500s are usually the slowest requests you have. If you want to distinguish them, keep them in the histogram but split them by a status attribute rather than dropping them.

Step 4 — Shape histogram buckets with a View. Default buckets rarely match a real latency profile, and a bucket layout that brackets your service's actual range is the difference between a usable p99 and a straight line. Attach a View when the provider is constructed — views are bound at provider construction, so a view added after the first record has no effect on that instrument.

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation

latency_view = View(
    instrument_name="http.server.duration",
    aggregation=ExplicitBucketHistogramAggregation(
        boundaries=[5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000]
    ),
)

provider = MeterProvider(metric_readers=[reader], views=[latency_view])
metrics.set_meter_provider(provider)

Quantiles are interpolated within a bucket, so precision comes from bucket density where the interesting mass sits, not from bucket count. If your p99 alert fires at 250 ms, make sure boundaries straddle it closely; a jump from 100 to 1000 makes that alert threshold unmeasurable.

Step 5 — Add an observable gauge for sampled state. For a value you read rather than increment, register a callback that yields one Observation per series. The reader invokes it on every collection. A gauge holds a last-value aggregation, so only the most recent sample per attribute set survives into the export; this is exactly what you want for instantaneous readings like pool depth or resident memory, where an average across the interval would hide spikes.

from opentelemetry.metrics import CallbackOptions, Observation

def read_pool(options: CallbackOptions):
    # Keep this fast and side-effect free; it runs on the reader thread.
    in_use = current_pool_usage()  # your own cheap accessor
    yield Observation(in_use, {"pool": "primary"})

meter.create_observable_gauge(
    "db.pool.in_use",
    callbacks=[read_pool],
    unit="{connection}",
    description="Connections currently checked out",
)

Reach for an ObservableCounter instead of a synchronous Counter only when the total already lives somewhere you can read, such as a kernel byte counter or a library's internal tally; in that case sampling it on each interval is correct and you avoid double counting. For discrete events your own code generates, the synchronous Counter is always the right tool because it cannot miss an event that occurs between two collections.

Choosing attributes

Attributes are what turn one instrument into many time series, and they are the single largest driver of metrics cost and query performance. Every distinct combination of attribute values on an instrument creates a separate series the backend must store and index. Attach attributes that you will group or filter by in dashboards and alerts, such as route, method, and status class, and never attach values that are effectively unique per event, such as user IDs, full URLs with query strings, or trace IDs. A useful test is to ask whether a value belongs on a chart axis or in a log line; chart-axis values are attributes, log-line values are not — put the high-cardinality detail on a span or in a structured log record instead, where storage is per-event rather than per-series.

When you cannot trust upstream code to stay disciplined, enforce the boundary centrally with a View that allow-lists attribute keys, so even if an instrument is called with a noisy attribute the SDK drops it before aggregation:

safe_attrs = View(
    instrument_name="http.server.request.count",
    attribute_keys={"http.route", "http.request.method", "http.status_class"},
)

Bucketing status codes into classes (2xx, 4xx, 5xx) rather than recording every numeric code is a common way to keep a useful dimension while bounding its growth; the same arithmetic and the mitigations for it are worked through in controlling label cardinality.

Recording from async and threaded code

Instruments are thread-safe and are meant to be shared: one module-level Counter called from every worker thread, task, or request is the intended pattern, and the SDK serializes the updates internally. Synchronous add and record calls do only in-memory aggregation, so they are safe to make from inside a coroutine without awaiting anything and without measurable event-loop impact.

Observable callbacks are the exception. They run on the metric reader's collection thread, not on your event loop, so a callback that awaits, queries a database, or takes a contended lock stalls collection for every instrument in the process — and a callback that overruns the reader's timeout gets its observations dropped for that cycle. Keep callbacks to reading an already-computed value, and refresh that value from a background task on your own schedule.

Configuration options

Verification

The fastest way to confirm an instrument is wired correctly is to collect it in-process rather than chasing it through a Collector. Attach an InMemoryMetricReader alongside your real reader in a test, record on each instrument once, and read the collected payload directly — no export interval to wait for and no network hop to debug.

from opentelemetry.sdk.metrics.export import InMemoryMetricReader

test_reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[test_reader], views=[latency_view])
metrics.set_meter_provider(provider)

request_counter.add(1, {"http.route": "/checkout", "http.status_code": 200})
latency.record(42.5, {"http.route": "/checkout", "http.status_code": 200})

data = test_reader.get_metrics_data()   # forces a collection, including callbacks

In a running service, call provider.force_flush() with a ConsoleMetricExporter attached instead. Either way you are checking three things: the counter carries the attribute series you expect, the histogram reflects your custom boundaries rather than the SDK defaults, and the gauge callback actually fired.

Expected Output:

{
  "metrics": [
    {
      "name": "http.server.request.count",
      "sum": {
        "isMonotonic": true,
        "dataPoints": [{
          "asInt": "1",
          "attributes": [
            {"key": "http.route", "value": {"stringValue": "/checkout"}},
            {"key": "http.status_code", "value": {"intValue": "200"}}
          ]
        }]
      }
    },
    {
      "name": "http.server.duration",
      "histogram": {
        "dataPoints": [{
          "count": "1",
          "sum": 42.5,
          "min": 42.5,
          "max": 42.5,
          "bucketCounts": ["0", "0", "0", "0", "1", "0", "0", "0", "0", "0", "0"],
          "explicitBounds": [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000]
        }]
      }
    },
    {
      "name": "db.pool.in_use",
      "gauge": {
        "dataPoints": [{"asInt": "7", "attributes": [{"key": "pool", "value": {"stringValue": "primary"}}]}]
      }
    }
  ]
}

The explicitBounds array is the assertion that matters most: if it shows the SDK defaults (0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000), your view did not attach. The single 1 in bucketCounts sits in the fifth slot, which is the 25 < v ≤ 50 bucket — the right home for a 42.5 ms measurement.

Common mistakes

• Error signature: a counter's series never moves and a warning mentions a negative amount. Root cause: Counter.add() was called with a negative value; a monotonic counter rejects negatives and leaves the series unchanged. Remediation: switch to an UpDownCounter, which accepts signed deltas, and reserve Counter for quantities that only grow.

• Error signature: the exported histogram carries the default explicitBounds no matter how you edit the boundary list. Root cause: the View never matched — its instrument_name differs from the histogram's name, or the view was added after the MeterProvider was built and the first record had already run. Remediation: match the instrument name exactly (wildcards are opt-in) and pass views=[...] at provider construction, before any measurement.

• Error signature: collection cycles get slower over time and unrelated instruments start missing export windows. Root cause: an observable callback does blocking work — a database query, an HTTP call, or a contended lock — on the reader thread, delaying every other instrument's collection. Remediation: read a cached in-memory value inside the callback and refresh it from a background task, keeping the callback to microseconds.

• Error signature: memory and backend series count climb steadily under load, and queries on the instrument slow down or get rejected. Root cause: an unbounded attribute value — a user ID, a raw path with IDs in it, a trace ID — is being passed on every call, so each event mints a new series. Remediation: normalise the value to a bounded set (route template instead of raw path, status class instead of status code) and add a View with attribute_keys as a guardrail so a future caller cannot reintroduce the problem.

Related

• The OpenTelemetry metrics SDK in Python — the parent reference covering the MeterProvider, readers, views, and temporality that these instruments feed.
• Exporting OTLP metrics to the Collector — the export hop that ships the aggregated values recorded here.
• Choosing between Counter, Gauge, Histogram, and Summary — the same instrument decision framed for a Prometheus-native pipeline.
• Controlling label cardinality in Prometheus — the arithmetic behind attribute choices and how to keep series counts bounded.
• OpenTelemetry vs Prometheus for Python metrics — push versus pull, and which instrumentation library a new service should start with.

Frequently Asked Questions