Python Observability

Securing OTLP with TLS and Headers

Telemetry carries request paths, user identifiers, error messages and occasionally things that should never have been in a span attribute at all. It deserves the same transport protections as any other data leaving a host — and, equally, the hop that never leaves the host deserves none of them. This page covers where to draw that line, how to get credentials into a collector without putting them in Python code, and the certificate failures that present as network problems. It is a task article under collector topology and deployment, part of the Python telemetry pipelines and delivery section.

Which hop needs what Three hops are drawn in sequence. The first, from the Python process to a collector on the same node or in the same pod, stays inside the host's network namespace and never touches a wire, so it needs no encryption and no credential; adding either would mean distributing and rotating certificates across every application. The second, from the agent to a gateway inside the same cluster, crosses the pod network, so it is encrypted with TLS and optionally authenticated with a mutual certificate where the cluster boundary is shared. The third, from the gateway to an external backend, crosses the internet and carries data from the entire fleet, so it uses TLS with a verified chain and an authorization header holding a token that exists in exactly one place. Beneath the hops is the count of components holding a secret in each arrangement: zero applications, zero agents, and three gateway replicas. encryption where it protects something, and nowhere else Python process plaintext agent TLS gateway TLS + token backend why the first hop is different never reaches a network loopback or node-local crosses the pod network one certificate per tier leaves the estate entirely carries the whole fleet's data components holding a secret: 0 applications · 0 agents · 3 gateway replicas
Each hop gets the treatment its exposure warrants. Encrypting the loopback hop costs certificate management in every service and protects against nothing.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

Implementation

Step 1 — Leave the local hop plaintext, deliberately. The SDK defaults to attempting a secure connection, so a local agent needs the insecure flag or an http:// endpoint. This reads like a downgrade and is not: the connection is to 127.0.0.1 or to the node's own address, so the bytes never traverse a network anyone can observe. Encrypting it would require issuing a certificate to every node, distributing the trust anchor to every application image, and rotating both — a substantial operational burden purchased entirely with no threat model.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Local agent: explicit about being plaintext, so nobody "fixes" it later.
exporter = OTLPSpanExporter(
    endpoint="http://127.0.0.1:4317",
    insecure=True,
    timeout=5,
)

Step 2 — Encrypt everything past the first hop. From the agent outward the traffic crosses a network and aggregates many services' data, so it gets TLS with a verified chain. The collector's TLS configuration names the authority it will trust, which should be explicit rather than inherited from the image's default bundle — an inherited trust store means the set of authorities you accept changes when the base image does.

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: false
      ca_file: /etc/otel/certs/cluster-ca.crt
      # mutual TLS, where the boundary warrants it
      cert_file: /etc/otel/certs/agent.crt
      key_file: /etc/otel/certs/agent.key

Step 3 — Supply credentials from the environment, never from code. A token in a Python file is a token in source control, in every image layer built from it, and in every developer's checkout. A token in an environment variable backed by a secret exists in the secret store and in the process's memory, which is the smallest footprint available without a dedicated fetching mechanism. The collector supports environment substitution directly, so the manifest references a secret and the configuration references the variable.

exporters:
  otlp/vendor:
    endpoint: ingest.vendor.example:443
    headers:
      authorization: "Bearer ${env:VENDOR_INGEST_TOKEN}"
    tls:
      insecure: false        # system trust store is correct for a public authority

Step 4 — Hold the credential in the tier with the fewest instances. This is the operational argument that decides the topology more often than any performance consideration. A token held by every application is copied into every deployment's secret reference, and rotating it means redeploying the fleet in a coordinated window. A token held by three gateway replicas is one secret, rotated by updating one object, with a rolling restart measured in seconds. The difference is not security in the cryptographic sense — the same secret, the same algorithm — but in whether rotation is a routine act or a project.

Step 5 — Verify the chain from inside the container that will use it. Most certificate failures are trust store differences, and they cannot be reproduced from a laptop. The check that matters runs in the actual image, against the actual endpoint, with the actual mounted bundle.

kubectl exec deploy/otel-gateway -c collector -- \
  openssl s_client -connect ingest.vendor.example:443 \
    -CAfile /etc/ssl/certs/ca-certificates.crt </dev/null 2>&1 | head -12

Expected Output: a verified chain, with the authority named.

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=0 CN = ingest.vendor.example
verify return:1
Verify return code: 0 (ok)
What a rotation costs in each topology Two rotation scenarios. In the first, the backend token lives in application configuration, so every one of forty services holds a reference to it. Rotating means updating the secret, then redeploying forty services, coordinating so that none is left holding the old value after it is revoked, and handling the services whose teams are unavailable that week. The elapsed time is measured in days and the failure mode is a service that quietly stops delivering telemetry. In the second, the token lives only on three gateway replicas. Rotating means updating one secret and restarting three pods, with a window of seconds during which both values are accepted by the backend. The comparison notes that the cryptographic properties are identical in both cases and only the operational cost differs, which is exactly why the topology decision is a security decision. rotating one token token in every application 40 deploys, coordinated, over days failure mode: one service quietly stops delivering token on the gateway only one secret updated, three pods restarted, seconds identical cryptography, completely different operations — which is why topology is a security decision
Nothing about the cryptography differs between these two arrangements. Everything about whether rotation actually happens does.

Why a working connection proves less than it looks

Certificate problems in telemetry pipelines have a characteristic shape: everything works in staging, the same configuration fails in production, and the error names a network condition rather than a trust one. Three mechanisms account for most of it.

The trust store is part of the image, not the configuration. A collector that validates a public certificate relies on the certificate bundle shipped in its base image. Change the base image — a routine upgrade, a distribution switch, a move to a distroless variant — and the set of authorities you trust changes with it. A backend whose certificate is signed by an authority present in one bundle and absent from another fails immediately after an upgrade that had nothing to do with telemetry, which makes the cause almost impossible to guess from the symptom.

An intermediate certificate can be missing without anyone noticing. A server presents its own certificate plus the intermediates needed to chain to a trusted root. If it omits an intermediate, clients that have cached that intermediate from previous connections succeed and clients starting fresh fail. A long-running collector can therefore work for months and fail on the first restart after the backend's certificate is reissued.

Hostname verification is stricter than reachability. A connection that reaches the right host with the wrong name in the certificate fails verification, which is correct, and reports it in language that reads like a connection failure. This is common when a service is addressed by an internal DNS alias, a pod IP, or a name that differs from the one in the certificate's subject alternative names — the connection is fine and the identity check is not.

The practical defence against all three is to verify from inside the deployed container rather than from anywhere convenient, and to treat the certificate authority as configuration rather than as an environmental default. Naming the authority explicitly, as step 2 does, converts a silent change into a failure at the moment the configuration changes, which is when somebody is watching.

One more consideration belongs here, because it is the reason several teams end up encrypting the local hop against their own better judgement: compliance frameworks that mandate encryption in transit, read literally, appear to cover a loopback connection. The productive response is to describe the hop accurately — a connection within one network namespace, or between two containers of one pod, that no other workload can observe — rather than to implement a control that adds certificate distribution to forty services and protects nothing. Where the framework genuinely requires it, a sidecar collector is the cheaper answer than a node agent, because the loopback hop inside a pod is easier to argue about than a hop across a node's interface.

Which protection answers which threat A table of threats to an OTLP connection and the protection that answers each. Someone reading telemetry in transit: TLS encryption. A client connecting to a fake collector: TLS with server certificate verification against a trusted CA. An unknown workload sending data into the pipeline: mutual TLS, or a bearer token in a header checked by the collector. A leaked token being replayed indefinitely: short-lived tokens and rotation. A valid client sending far too much: rate limits and the memory limiter at the receiver. The note says encryption alone does not authenticate anyone, and authentication alone does not encrypt. threat protection reading telemetry in transit TLS encryption a fake collector server certificate verification an unknown sender mTLS, or a bearer token header a leaked token replayed short-lived tokens, rotation a valid client flooding rate limits, memory limiter encryption does not authenticate anyone; authentication does not encrypt
TLS and credentials solve different problems. A secure OTLP link needs both, plus limits for clients that are valid but noisy.

Configuration options

Setting Local hop Cluster hop External hop
TLS off on on
Trust anchor none explicit ca_file system or explicit
Client certificate none optional mutual TLS rarely
Credential none none bearer token header
Where the secret lives nowhere nowhere gateway secret
Failure to expect none certificate rotation trust store drift

Verification

Check the two properties independently, because a single test that passes hides which one is doing the work.

# 1. Is the local hop actually local and plaintext?
kubectl exec deploy/checkout -c app -- python -c "
import os; print(os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'])"

# 2. Does the external hop authenticate, and does it reject a bad token?
kubectl exec deploy/otel-gateway -- sh -c '
  curl -s -o /dev/null -w "%{http_code}\n" -X POST \
    -H "authorization: Bearer $VENDOR_INGEST_TOKEN" \
    https://ingest.vendor.example/v1/traces
  curl -s -o /dev/null -w "%{http_code}\n" -X POST \
    -H "authorization: Bearer wrong" \
    https://ingest.vendor.example/v1/traces'

Expected Output: a local endpoint, an accepted request, and a rejected one — the third line being the part that proves authentication is actually enforced.

http://127.0.0.1:4317
200
401

Common mistakes

Enabling TLS on the loopback hop. Error signature: certificate errors in application startup, in every service, whenever a node certificate rotates. Root cause: encrypting a hop that never leaves the host. Remediation: use an explicit http:// endpoint with the insecure flag, and document why.

A token in application configuration. Error signature: a rotation that takes a week, and a service found weeks later still holding the revoked value. Root cause: the credential distributed to the tier with the most instances. Remediation: move the authenticated export to the gateway and remove the secret reference from application manifests.

Relying on the image's default trust store. Error signature: certificate verification failures immediately after an unrelated base image upgrade. Root cause: the trusted authority set is inherited rather than declared. Remediation: mount and name the certificate authority explicitly in the collector's TLS configuration.

Verifying from the wrong place. Error signature: a check that passes from a workstation and fails in the cluster. Root cause: different trust stores, different DNS, different egress path. Remediation: run the verification inside the deployed container, as in step 5.

Confusing a name mismatch with a connectivity problem. Error signature: a verification failure on an endpoint that responds to a plain connection test. Root cause: the certificate's subject alternative names do not include the address being used. Remediation: address the endpoint by a name the certificate covers, or reissue the certificate to cover the name in use.

Frequently Asked Questions

Does the application-to-collector hop need TLS?

Not when the collector is on the same node or in the same pod. The traffic stays inside the host, so encryption protects against nothing while adding certificate distribution and rotation to every application. Encrypt the hop that leaves the host instead.

Where should the backend token live?

In the collector tier with the fewest instances, read from an environment variable backed by a secret. Putting it in application configuration means every service holds a copy of a credential it does not need, and rotating it becomes a fleet-wide deploy.

Why does my OTLP exporter fail with a certificate error only in production?

Almost always a trust store difference. The image, the base distribution or the mounted certificate bundle differs between environments, so the chain presented by the backend validates in one and not the other. Check which authority signed the certificate and whether that authority is in the container's trust store, not whether the connection reaches the host.

Should I use mutual TLS between collector tiers?

It is worth it when the agent-to-gateway hop crosses a boundary somebody else administers, and overkill inside one cluster where a network policy already restricts who can reach the gateway. The cost is certificate issuance and rotation for every agent.

Related Guides