Python Observability

Metrics from Short-Lived Jobs with the Pushgateway

A scrape-based metrics system asks a process for its current values on an interval. A job that runs for four seconds is almost never up when that happens, so its metrics endpoint is read approximately never. This page covers pushing instead: the registry, the grouping key that decides what is overwritten, the timestamp that makes alerting possible, and the several ways this component is misused. It is a task article under telemetry from serverless and batch Python, part of the Python telemetry pipelines and delivery section.

Why the scrape never happens A fifteen second scrape interval is drawn as regular marks along a timeline. Below it, six runs of a job that takes four seconds each occur at various points, and only one of them happens to overlap a scrape mark. The five that do not overlap produce no metrics at all, despite exposing a perfectly correct endpoint throughout their lives. Beneath that, the same six runs are drawn pushing their final values to a gateway at the moment each completes; the gateway holds those values and is scraped on the normal interval, so every run is represented. The property noted alongside is that the gateway continues serving the last values after the job stops running entirely, which is why alerting must be based on the age of a success timestamp rather than on the presence of the metric. a 4-second job against a 15-second scrape scrape runs five of six runs are never scraped — their endpoints were perfect and unread pushing instead gateway holds the last values — and keeps holding them after the job stops running which is why the alert is on the age of a timestamp, never on the metric being present
The endpoint was never the problem. The scrape was, and pushing moves the values somewhere that is up when the scrape arrives.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0"

export PUSHGATEWAY_URL="http://pushgateway.observability.svc:9091"
export JOB_NAME="nightly-reconciliation"

Implementation

Step 1 — Use a registry local to the job. The default registry accumulates whatever any imported library has registered, including platform collectors describing the Python process. Pushing all of that means the gateway serves a job's process metrics forever after it exits, which is both misleading and a source of series nobody wants. A local registry contains exactly what you put in it.

from prometheus_client import CollectorRegistry, Counter, Gauge

registry = CollectorRegistry()          # NOT the default registry

rows_processed = Counter(
    "job_rows_processed_total", "Rows processed by this run",
    registry=registry)
rows_rejected = Counter(
    "job_rows_rejected_total", "Rows rejected by this run",
    registry=registry)
duration = Gauge(
    "job_duration_seconds", "Wall-clock duration of the run",
    registry=registry)
last_success = Gauge(
    "job_last_success_timestamp_seconds", "When this job last completed successfully",
    registry=registry)

Step 2 — Choose a grouping key that identifies the job. The gateway stores pushed metrics under a grouping key, and a push replaces everything under that key. A key of job name alone works for a singleton job; job name plus a stable instance works for a job that runs in several places. Including a run identifier or a timestamp in the key is the classic mistake: every run creates a new permanent group, and the gateway accumulates them without limit.

Step 3 — Push once, at the end, with a success timestamp. The timestamp is what makes alerting possible. Because the gateway keeps serving the last values indefinitely, no metric goes missing when a job stops running — the only observable change is that a timestamp stops advancing.

import time
from prometheus_client import push_to_gateway

started = time.time()
try:
    result = run_reconciliation()
    rows_processed.inc(result.rows)
    rows_rejected.inc(result.rejected)
finally:
    duration.set(time.time() - started)
    last_success.set_to_current_time()      # only on the success path in real code
    push_to_gateway(
        os.environ["PUSHGATEWAY_URL"],
        job=os.environ["JOB_NAME"],
        registry=registry,
        grouping_key={"instance": os.environ.get("SHARD", "0")},
    )

Expected Output: the gateway serving the job's values, ready for the next ordinary scrape.

# TYPE job_rows_processed_total counter
job_rows_processed_total{job="nightly-reconciliation",instance="0"} 100000
job_rows_rejected_total{job="nightly-reconciliation",instance="0"} 158
job_duration_seconds{job="nightly-reconciliation",instance="0"} 41.208
job_last_success_timestamp_seconds{job="nightly-reconciliation",instance="0"} 1.789e+09

Step 4 — Use replace semantics, not add. A push with the same grouping key replaces the previous values, which is what you want: a rerun corrects the record. The additive variant exists for the case where several independent producers contribute to one group, and using it by mistake produces counters that climb without bound across runs and a duration that is the sum of every run ever.

Step 5 — Delete the group when the job is retired. Because the gateway serves values forever, a decommissioned job continues to report a successful run from whenever it last executed. Anything alerting on the age of that timestamp will fire eventually — which is the good outcome — but a dashboard showing the job as healthy is actively misleading in the meantime.

# remove a retired job's group explicitly
curl -X DELETE "$PUSHGATEWAY_URL/metrics/job/nightly-reconciliation/instance/0"

Step 6 — Alert on the timestamp, never on the metric. This is the consequence of everything above. job_last_success_timestamp_seconds aged past the expected interval is the only reliable signal that a job is not running, because every other metric keeps its last value indefinitely.

The grouping key decides whether anything is ever removed Two grouping key designs are compared over a month of hourly runs. The first includes a run identifier in the key, so every one of the seven hundred and twenty runs creates a distinct group that the gateway keeps forever; the gateway's own memory grows without bound, the scrape it serves grows with it, and the metrics store gains a new set of series every hour. The second uses the job name and a stable shard identifier, so all seven hundred and twenty runs write to the same group and each one replaces the last; the gateway holds exactly one group, its scrape is constant in size, and the metrics store sees one series per metric whose value changes hourly. The note observes that the first design is usually chosen because it feels like it preserves history, and that the history belongs in the metrics store rather than in the gateway. 720 hourly runs, two grouping keys key includes the run id … 720 groups, none of which is ever removed gateway memory grows · its scrape grows · new series every hour key is job name + shard one group · each run replaces the last gateway constant · one series per metric · history lives in the metrics store, where it belongs the first design is chosen to preserve history — and preserves it in the one place that cannot query it
The gateway is a staging area, not a store. A grouping key that treats it as a store fills it with data nothing can query.

When to use something else

The Pushgateway solves a specific problem and is frequently reached for when a different tool fits better. Three alternatives deserve consideration first.

Span attributes. If the job is already producing a trace — and it should be, per telemetry from serverless and batch Python — then the counts can live as attributes on the run span. There is no extra component, the numbers are attached to the run that produced them rather than to a label set, and high-cardinality values such as a run identifier are free. The cost is that deriving a time series from them depends on the backend supporting it.

Structured log records. A single completion record carrying the counts is the simplest possible mechanism, works with any log pipeline, and is trivially queryable. Many teams find that a job's numbers are consulted a handful of times a month, in which case a log query is entirely adequate and the metrics pipeline adds nothing.

The OpenTelemetry metrics SDK with a push exporter. The metrics SDK pushes by design, so a job that exports OTLP metrics with an explicit flush at the end needs no gateway at all. This is the natural choice for a fleet already sending OTLP, and it avoids the grouping key semantics entirely, since the collector applies its normal aggregation.

The case where the gateway is genuinely the right answer is narrow but real: an existing Prometheus deployment is the only consumer, the numbers must be a proper time series, and adding an OTLP path for one job is disproportionate. Recognising that case rather than defaulting to it saves a component and a class of subtle failures.

The properties that surprise people

Three behaviours of this component account for nearly every support question about it, and all three follow from one design decision: it is a cache of last-known values, not a time series database.

It does not record history. Pushing a value every hour does not create an hourly series inside the gateway; it overwrites. The series exists only because Prometheus scrapes the gateway on its own interval and stores each observation. If the scrape interval is longer than the job's period, runs are silently skipped — two runs between scrapes means the first is never recorded anywhere. Matching the scrape interval to the job frequency, or accepting that only the most recent run is captured, is a decision to make deliberately.

It has no notion of the job being over. A metric pushed once is served until something deletes it. There is no expiry, no staleness marker and no way for the gateway to know that a job has been decommissioned. This is why the success timestamp is not an optional extra: it is the only mechanism by which absence becomes observable.

Its labels override the scrape's. Metrics served by the gateway carry the job and instance labels from the grouping key, and Prometheus must be configured with honour-labels enabled for those to survive the scrape rather than being replaced with the gateway's own target labels. Getting this wrong produces metrics that all appear to come from the gateway itself, which makes them impossible to attribute to the job that produced them — a configuration error that looks like an instrumentation error.

What to push and what not to A table of metrics a batch job might push to the Pushgateway, with guidance on each. The timestamp of the last successful completion, as a gauge: push it; it is the single most useful job metric. The run duration, as a gauge: push it. Items processed in the last run, as a gauge: push it. A per-item latency histogram: usually do not; it describes only the last run and is overwritten. Per-instance metrics from many parallel workers under one grouping key: do not; they overwrite each other. The note says the Pushgateway holds the last value per grouping key, so it suits a few per-run facts, not continuous series. metric push it? last success timestamp (gauge) yes — the most useful one run duration (gauge) yes items processed (gauge) yes per-item latency histogram rarely — overwritten each run parallel workers, one group key no — they overwrite each other the Pushgateway keeps the last value per grouping key it suits a few per-run facts, not continuous series
Push a few facts about each run. The gateway remembers only the latest value, so anything else is overwritten.

Configuration options

Decision Recommended Failure if wrong
Registry a local CollectorRegistry platform metrics served forever
Grouping key job name plus stable instance unbounded group accumulation
Push mode replace counters that climb across runs
Success timestamp always pushed no way to alert on absence
Deletion on retirement explicit DELETE a dead job reporting health
Alert basis timestamp age an alert that can never fire
Push timing once, at the end partial values from a failed run

Verification

Confirm the gateway holds exactly one group per job, and that the timestamp advances.

# what the gateway is currently serving
curl -s "$PUSHGATEWAY_URL/metrics" | grep -E '^job_(last_success|duration)'

# how many distinct groups exist — should equal the number of jobs, not runs
curl -s "$PUSHGATEWAY_URL/api/v1/metrics" \
  | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["data"]))'

Expected Output: one group per job, with a timestamp from the most recent run.

job_last_success_timestamp_seconds{job="nightly-reconciliation",instance="0"} 1.789e+09
job_duration_seconds{job="nightly-reconciliation",instance="0"} 41.208
4

A group count that grows between checks means the grouping key includes something that varies per run, which is the failure from step 2 and gets expensive quickly.

Common mistakes

Pushing the default registry. Error signature: process metrics from a job that exited hours ago, served as current. Root cause: the default registry includes platform collectors. Remediation: build a registry containing only the job's own metrics.

A run identifier in the grouping key. Error signature: a gateway whose memory and scrape size grow every hour. Root cause: every run creating a permanent group. Remediation: job name plus a stable instance; the run identifier belongs on a span or a log record.

Alerting on the metric rather than its age. Error signature: an alert that never fires, for a job that has not run in a month. Root cause: the gateway serving stale values indefinitely. Remediation: alert on the age of the success timestamp.

Using the additive push mode. Error signature: a duration metric of several thousand seconds for a job that takes forty. Root cause: values added to the existing group rather than replacing it. Remediation: use the replacing push, which is the default in the client library.

Never deleting retired jobs. Error signature: a dashboard showing healthy jobs that no longer exist. Root cause: groups persisting after decommissioning. Remediation: delete the grouping key as part of retiring the job, and audit the gateway's group list periodically.

Frequently Asked Questions

Why not just expose a metrics endpoint from the job?

Because nothing will scrape it. A scrape happens on an interval, and a job that runs for four seconds will almost never be up when one occurs. The endpoint exists, is never read, and produces the impression of instrumentation without any of its effect.

What is the grouping key and why does it matter?

It is the set of labels that identifies a group of pushed metrics in the gateway. A push replaces everything under the same key, so a key that includes a run identifier accumulates one group per run forever, while a key of job name plus a stable instance overwrites cleanly.

Do pushed metrics disappear when the job finishes?

No, and that is the property to understand. The gateway keeps serving the last pushed values indefinitely, so a job that stopped running entirely continues to report its final success. Alerting must therefore be on the age of the success timestamp, not on the value itself.

Is the Pushgateway the only option?

No. Recording the job's numbers as span attributes or structured log fields and deriving metrics downstream avoids the extra component entirely, and keeps the numbers attached to the run that produced them. The gateway is the right answer when an existing Prometheus deployment is the only consumer.

Related Guides