Securing the Metrics Endpoint

The default way to expose Prometheus metrics from a Python service is a /metrics route on the application itself — and the application is usually reachable from the internet. A public metrics endpoint tells anyone who asks which routes exist, how much traffic each receives, which dependencies the service calls, and which library versions it runs. This article covers moving the endpoint off the public path, restricting and authenticating access to it, encrypting the scrape, and checking the result. It belongs to Prometheus client instrumentation in the Python metrics and instrumentation section.

Prerequisites

A Python service exposing Prometheus metrics, as in exposing custom metrics with prometheus_client, and control over its deployment manifest and the Prometheus scrape configuration.

Implementation steps

Step 1 — Serve metrics on a separate port. The single most effective change. The public ingress routes to the application port; a second port, used only for metrics, is never added to any ingress or public load balancer.

# single-process service
from prometheus_client import start_http_server
start_http_server(9464, addr="0.0.0.0")      # metrics only; the app stays on 8000

For an ASGI service where running a second listener is inconvenient, a small separate ASGI app served by a second Uvicorn server in the same process works too; the essential property is a port the public path cannot reach. With multiprocess servers, the separate listener is best started once, from the master process, with a multiprocess registry — the approach in Prometheus multiprocess mode with Gunicorn.

Step 2 — Restrict the port to the scrapers. In Kubernetes, a network policy allows the metrics port only from the monitoring namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: orders-api-metrics}
spec:
  podSelector: {matchLabels: {app: orders-api}}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}
      ports: [{port: 9464, protocol: TCP}]
    - ports: [{port: 8000, protocol: TCP}]

Outside Kubernetes, a host firewall rule or a security group that admits the metrics port only from the Prometheus hosts is the equivalent.

Step 3 — Require a token. A static bearer token, stored as a secret and compared in constant time, stops anything inside the network that is not the scraper.

import hmac, os
from wsgiref.simple_server import make_server
from prometheus_client import make_wsgi_app

TOKEN = os.environ["METRICS_TOKEN"].encode()
metrics_app = make_wsgi_app()

def guarded(environ, start_response):
    supplied = environ.get("HTTP_AUTHORIZATION", "").removeprefix("Bearer ").encode()
    if not hmac.compare_digest(supplied, TOKEN):
        start_response("401 Unauthorized", [("Content-Type", "text/plain")])
        return [b"unauthorized\n"]
    return metrics_app(environ, start_response)

make_server("0.0.0.0", 9464, guarded).serve_forever()     # run in a daemon thread

# prometheus scrape job
- job_name: orders-api
  authorization:
    type: Bearer
    credentials_file: /etc/prometheus/secrets/orders-api-token
  kubernetes_sd_configs: [{role: pod}]

Step 4 — Encrypt the scrape if the network is not trusted. Inside a service mesh with mutual TLS, the scrape is already encrypted. Otherwise, start_http_server accepts a certificate and key, and the scrape job sets scheme: https with a CA to verify against. A bearer token sent over plain HTTP on an untrusted network can be captured and replayed.

start_http_server(9464, certfile="/tls/tls.crt", keyfile="/tls/tls.key")

What not to put in metrics in the first place

Securing the endpoint limits who can read the exposition; keeping sensitive values out of it limits the damage if the securing fails. Label values are the usual leak. A label containing a customer identifier, an email domain, a tenant name or an internal hostname puts that value in plain text on every scrape and in the metrics store for its retention period — often longer than logs are kept.

The rules that keep cardinality under control, described in controlling label cardinality in Prometheus, mostly keep sensitive values out too: bounded labels drawn from code rather than from requests. The exceptions are small, bounded, sensitive sets — a list of enterprise tenant names, say — which are safe for cardinality and still inappropriate in an exposition that many teams can read. Mapping them to opaque identifiers, or aggregating them into tiers, keeps the metric useful and the names private.

Info metrics deserve a second look as well. A build_info metric with the version and commit is useful for correlating deploys with changes in behaviour; one that also includes the full dependency list, environment variables or configuration file paths says more than the monitoring system needs. Keep info metrics to what dashboards actually use.

The same thinking applies to the logs and spans next to the metrics — logging personal data safely covers the equivalent for log records.

When the service is scraped through a proxy

Some platforms scrape through a sidecar or node agent rather than directly, and some managed Prometheus services pull through a gateway. In those setups, the endpoint needs to be reachable by the agent, and the agent is what must be restricted. The pattern is the same — a port that only the agent can reach — and the token, if used, is configured in the agent rather than in Prometheus.

The OpenTelemetry Collector's Prometheus receiver is a common example: it scrapes the service locally, then exports over OTLP with its own authentication. The metrics port can then be bound to 127.0.0.1 inside a pod shared with the collector sidecar, which removes it from the network entirely.

The endpoint's cost as an attack surface

A metrics scrape is not free. Rendering the exposition walks every metric and every label combination, formats each value as text, and — in multiprocess mode — opens and reads every process's files. For a service with a few thousand series, that is a few milliseconds of CPU; for one with a hundred thousand series and months of worker recycling, it can be hundreds of milliseconds, and it runs in the same process that serves users.

This makes a public metrics endpoint a cheap denial-of-service vector: no authentication, no rate limit, and a response that costs far more to produce than the request costs to send. Even without malice, a misconfigured second Prometheus, an uptime checker pointed at the wrong path, or a developer's dashboard polling every second can add measurable load.

The separate port removes the public vector. For the internal ones, two habits help. Watching the scrape duration Prometheus records for each target — scrape_duration_seconds — catches expositions that have grown expensive, usually because of a cardinality problem that deserves fixing anyway. And counting requests to the metrics endpoint itself, by client address if the service can see it, shows when something other than the expected scrapers is calling.

Configuration options

Verification

From outside the cluster, through the public hostname:

curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/metrics

Expected Output: a 404 from the application, since the route no longer exists on the public port.

404

From a pod outside the monitoring namespace, a request to the pod's metrics port should time out, which confirms the network policy. From Prometheus's targets page, the job should show every target as up, which confirms the scrape still works with the token and TLS. A target showing 401 means the token file is missing or wrong; a TLS error means the CA does not match the certificate.

Common mistakes

Metrics on the application port. Error signature: /metrics served through the public hostname. Root cause: the default route added to the main app. Remediation: a separate port.

Token compared with ==. Error signature: none visible, which is the problem. Root cause: a comparison whose time depends on how much of the token matches. Remediation: hmac.compare_digest.

Metrics auth tied to the application's auth service. Error signature: scrapes failing whenever the auth service is degraded. Root cause: an observability path depending on the thing it observes. Remediation: a static scrape token.

Token over plain HTTP on a shared network. Error signature: none visible. Root cause: a replayable secret in cleartext. Remediation: TLS or a mesh.

Sensitive label values. Error signature: tenant names or emails in the metrics store. Root cause: request data used as labels. Remediation: bounded, opaque labels from code.

Token in the scrape config as plain text. Error signature: the secret visible in the Prometheus configuration repository. Root cause: credentials used instead of credentials_file. Remediation: mount the token from a secret and reference the file.

No watch on scrape duration. Error signature: a service whose p99 latency rises every fifteen seconds. Root cause: an exposition grown expensive enough to stall request handling. Remediation: alert on scrape_duration_seconds and fix the cardinality behind it.

Frequently Asked Questions