Choosing Between Counter, Gauge, Histogram, and Summary

Picking the wrong Prometheus metric type produces graphs that lie: counters read as gauges drop to zero on every deploy, and summaries chosen for fleet-wide objectives return per-process nonsense that no query can repair. This walkthrough is for backend engineers and SREs instrumenting a Python service who need to settle the type question once, at design time, before the metric reaches a dashboard. It sits under metric types and cardinality control and is part of the broader Python metrics and instrumentation reference; once the type is settled, the second half of the design decision is keeping the label dimensions bounded, which is covered in controlling label cardinality in Prometheus.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0"

Python 3.10 or newer and a Prometheus server (>=2.50,<3.0) scraping the exposition endpoint are assumed. No environment variables are required for these examples beyond an open exposition port, which start_http_server provides; the endpoint mechanics themselves are covered in exposing custom metrics with prometheus_client.

The four types in one sentence each

A Counter is a value that only goes up and resets to zero on process restart; it answers how many events have occurred and is meaningful only as a rate. A Gauge is a value that can move in either direction and represents a snapshot of current state. A Histogram observes values into fixed cumulative buckets and exports the raw bucket counts so the server can derive quantiles at query time. A Summary computes a distribution summary inside the client process and exports the result directly.

The four split cleanly into two pairs. Counters and gauges describe a single number and differ only in whether that number can fall. Histograms and summaries describe a distribution of many numbers and differ in where the statistics are computed — on the server, from raw buckets, or in the process, before export. Almost every wrong choice in production comes from confusing one member of a pair with the other, so the questions below separate the pairs first and then resolve within each pair.

Implementation

The choice reduces to three questions answered in order, mirroring the decision tree above.

Step 1 — Does the value only ever increase, resetting only on restart? If so, it is a Counter. Counts of events — requests, errors, bytes processed, retries — are the canonical case. Never read a counter's raw value in a query; wrap it in rate() so process restarts are handled as resets rather than data loss.

from prometheus_client import Counter

# The _total suffix is the exposition convention; the client normalises it either way.
ERRORS = Counter("app_errors_total", "Errors raised", labelnames=("kind",))
ERRORS.labels(kind="timeout").inc()      # monotonic; always query through rate()

A common trap at this step is reaching for a Gauge because the number "goes up over time and I want the total." If the underlying events are discrete and you care about the rate or the increase over a window, it is still a Counter even though the displayed total grows. The test is not whether the displayed number increases but whether the instrument can ever decrease for a reason other than a restart. Error totals, requests served, and bytes written never legitimately decrease, so they are counters regardless of how you plan to visualise them. Keep the kind label drawn from a fixed, enumerable set — a counter is exactly as expensive as the product of its label values.

Step 2 — Is it a current value that moves both up and down? That is a Gauge: queue depth, in-flight requests, connection pool size, temperature, memory in use. Use set, inc, and dec, or track_inprogress() as a context manager for concurrency counts. For a value that is expensive to poll, register a callback with set_function so the value is computed at scrape time instead of on a hot path.

from prometheus_client import Gauge

QUEUE_DEPTH = Gauge("task_queue_depth", "Pending tasks")
QUEUE_DEPTH.set(get_queue_length())          # snapshot; can rise and fall

IN_FLIGHT = Gauge("http_requests_in_flight", "Requests currently being served")
with IN_FLIGHT.track_inprogress():           # inc on entry, dec on exit, exception-safe
    serve_request()

Gauges carry a subtlety in aggregation: because a gauge is a point-in-time value, summing it across replicas is only meaningful when the quantity is genuinely additive, such as total memory used by a fleet. Averaging or taking the maximum is correct for quantities like utilisation percentages. Choosing the wrong aggregation function over a gauge produces a number that is valid PromQL but semantically meaningless, so decide at design time how the gauge will be combined across instances. The same decision resurfaces inside a single service when it runs under Gunicorn or uWSGI: in multiprocess mode a gauge must declare multiprocess_mode so the collector knows whether to sum, average, or take the live minimum or maximum across worker processes.

Step 3 — Is it a distribution of observed values you want quantiles or a spread of? Then it is a Histogram or a Summary, and the deciding sub-question is aggregation. If you need quantiles across many replicas — almost every latency objective — choose a Histogram, because its buckets sum across instances and histogram_quantile() runs on the merged result.

from prometheus_client import Histogram

LATENCY = Histogram(
    "request_duration_seconds", "Request latency",
    # Explicit edges that straddle the 250 ms objective; +Inf is appended for you.
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
with LATENCY.time():                          # observes elapsed seconds on exit
    serve_request()

from prometheus_client import Summary

# Only for a single-instance distribution you will never aggregate.
GC_PAUSE = Summary("gc_pause_seconds", "GC pause duration")
with GC_PAUSE.time():
    run_gc_cycle()

A worked counterexample for the pairs above: for a cache hit ratio, resist a single Gauge holding the percentage. Expose two Counters — hits and total lookups — and divide their rates at query time with rate(cache_hits_total[5m]) / rate(cache_lookups_total[5m]). This preserves the correct windowing and aggregation that a precomputed gauge throws away, and it lets one instrumentation decision serve a five-minute panel and a thirty-day report.

Why the histogram usually wins

The histogram-versus-summary decision deserves its own scrutiny because it is the most consequential and the most often gotten wrong. A histogram records nothing more than a count per bucket, so the work it does at observe time is a cheap increment on a handful of counters. All the statistical effort — interpolating a quantile between two boundaries — happens later on the server, over whatever set of series the query selects, which is exactly why buckets can be summed across replicas before histogram_quantile() runs.

A summary instead computes its statistics inside the client, producing values tied irrevocably to one process. Precomputed quantiles cannot be averaged or summed: the p99 of two processes is not the average of their individual p99s, and PromQL has no way to recover the true combined quantile from the published ones. For any objective measured across more than one replica, that property alone forces the histogram.

There is a second, Python-specific reason the summary is nearly always the wrong answer. Unlike the Go client, prometheus_client implements no streaming quantile estimator at all: a Summary exports only _sum and _count, so the single statistic you can derive from it is an average. If you want any quantile from a Python process — even from one process you will never aggregate — you need a Histogram. That leaves the Summary useful only where an average and a rate of observations genuinely answer the question, such as a background daemon's mean work-item duration.

A practical note on histogram evolution: bucket boundaries are part of the data, not just the query. If you redeploy with new boundaries, samples recorded under the old boundaries keep them, and a quantile spanning the change blends two resolutions. Plan bucket layouts to outlast the dashboards that consume them, keep the count modest — each boundary is another stored series per label combination — and place an edge on every threshold you alert against so the interpolation has resolution exactly where it matters. The default bucket set spans 5 ms to 10 s and is a reasonable starting point for HTTP handlers, but it is rarely the right set for a queue consumer or a batch step.

The same reasoning transfers directly to the OpenTelemetry instrument names: Counter and UpDownCounter cover the monotonic and bidirectional single values, ObservableGauge covers the callback-polled snapshot, and Histogram covers the distribution. There is no summary instrument in the OTel data model at all, which is the standards body making the same judgement this section reaches — see recording counters and histograms with OpenTelemetry for the API shape, and OpenTelemetry vs Prometheus for Python metrics if you have not yet settled which pipeline records them.

Configuration options

Two further knobs apply to all four. namespace and subsystem prefix the metric name at construction time, which is the cleanest way to keep one service's series from colliding with another's in a shared Prometheus. Setting PROMETHEUS_DISABLE_CREATED_SERIES=true suppresses the _created timestamp series that counters, histograms, and summaries emit by default, cutting the stored series count for those instruments without changing any value you actually query.

Verification

Scrape the exposition endpoint and confirm each instrument renders with its declared # TYPE line and the expected series shape: counters and gauges as a single sample, histograms as _bucket plus _sum and _count, summaries as _sum and _count only. If a series shape does not match the type you intended, the instrument is wrong in code — no dashboard change will fix it.

curl -s localhost:8000/metrics | grep -E "app_errors|task_queue_depth|request_duration|gc_pause"

Expected Output: the four types appear distinctly in the exposition text.

# TYPE app_errors_total counter
app_errors_total{kind="timeout"} 1.0
# TYPE task_queue_depth gauge
task_queue_depth 7.0
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.25"} 1.0
request_duration_seconds_sum 0.031
request_duration_seconds_count 1.0
# TYPE gc_pause_seconds summary
gc_pause_seconds_sum 0.004
gc_pause_seconds_count 1.0

Note what is absent: the summary emits no quantile series, confirming that a p99 panel built on it would have nothing to read. The histogram, by contrast, exposes one _bucket series per boundary, and histogram_quantile(0.99, sum by (le) (rate(request_duration_seconds_bucket[5m]))) returns a fleet-wide answer from them. Run that query against a single replica and then against the whole job; both should return a plausible latency, and the fleet number should sit between the per-replica extremes.

Common mistakes

• Error signature: a counter panel sawtooths to zero on every deployment. Root cause: the query reads the raw cumulative value, which resets when the process restarts. Remediation: wrap counters in rate() or increase(), both of which detect the reset and treat it as a restart rather than a negative jump.

• Error signature: a p99 panel spanning replicas shows a suspiciously smooth line, a per-instance value, or nothing at all. Root cause: the instrument is a Summary, whose statistics are computed per process and — in Python — do not include quantiles to begin with. Remediation: switch to a Histogram and compute the quantile from summed buckets with histogram_quantile().

• Error signature: a ratio panel cannot be re-windowed or broken down by instance without re-instrumenting. Root cause: the division happened in the application and was stored as one Gauge, discarding the numerator and denominator. Remediation: expose the two underlying Counters and divide their rates in the query, which keeps every window and grouping available afterwards.

• Error signature: histogram_quantile() returns a value pinned exactly to a bucket edge, or jumps between two fixed numbers as load changes. Root cause: the objective threshold falls in a wide gap between boundaries, so the interpolation has no resolution where it matters. Remediation: add explicit edges around the threshold, redeploy, and be aware that only samples recorded after the change carry the finer resolution.

A quick reference for the common cases

To make the decision automatic for the measurements that appear in almost every service, fix these defaults in mind. Request and error totals are Counters, queried with rate(). In-flight requests, queue depth, and connection-pool size are Gauges. Request latency and response size are Histograms with buckets aligned to the objective — the pattern used throughout instrumenting Flask with Prometheus metrics. Cache hit ratio is two Counters divided at query time, never a Gauge. A mean garbage-collection pause on a single daemon is the rare defensible Summary. Anchoring on these defaults removes most of the per-metric deliberation and leaves only the genuinely novel measurements to reason about from the three questions above.

When a measurement does not fit any default, walk the questions in order — only increases, moves both ways, or a distribution — and let the answer pick the type before you think about labels. Then apply the cardinality lens: if the natural label is unbounded, keep that dimension in traces or logs and instrument the bounded view instead. The type decision and the label decision together produce telemetry that is cheap to store and correct to query, and they are far easier to get right at design time than to unwind after a dashboard depends on them.

Related

• Metric types and cardinality control — the parent reference covering instrument choice and label budgets together.
• Controlling label cardinality in Prometheus — the second half of the design decision: bounding the label dimensions of the type you just chose.
• Exposing custom metrics with prometheus_client — registry, exposition endpoint, and multiprocess mode for the instruments defined here.
• Recording counters and histograms with OpenTelemetry — the same type decision expressed in the OpenTelemetry instrument API.
• OpenTelemetry vs Prometheus for Python metrics — choosing the pipeline that will carry these metric types to storage.

Frequently Asked Questions