Agent vs Gateway Collector Deployment

Both tiers run the same binary with the same configuration language, which makes it easy to assume they are interchangeable. They are not: their value comes from where they sit, and each can do something the other structurally cannot. This page sets out the differences that matter in practice and the order to adopt them in. It is a task article under collector topology and deployment, part of the Python telemetry pipelines and delivery section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

The collector itself is deployed by the platform; the application needs only an endpoint.

Implementation

Step 1 — Deploy the agent first, and measure what it changed. The agent's contribution is easy to state and easy to verify: the application's export latency should fall to single-digit milliseconds and its export failure rate to zero, because the hop no longer leaves the node. Both are visible in the SDK's own numbers, and both are worth recording before and after, because they are the evidence that justifies the tier when somebody asks what it is for.

Step 2 — Keep the agent's processor list short. An agent runs once per node and therefore once per every pod's worth of traffic, so anything expensive in it is paid many times. The processors that belong there are the cheap, per-record ones: resource detection, which reads the environment once at start; batching, which is pure win; and a memory limiter, which is protection rather than work. Anything that needs to look across records — aggregation, trace-level policy, deduplication — is both more expensive and structurally wrong at this tier.

Step 3 — Introduce the gateway when a fleet-wide capability is required. There are three such capabilities and it is worth being strict about the list, because a gateway adopted without one is a component to run, upgrade and be paged for, in exchange for nothing.

# gateway — the three things only this tier can do
processors:
  tail_sampling:                 # 1. needs the whole trace
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
exporters:
  otlp/primary:                  # 2. fan out, changed without a deploy
    endpoint: ingest.vendor.example:443
    headers: { authorization: "Bearer ${env:VENDOR_TOKEN}" }   # 3. one place
  otlp/archive:
    endpoint: archive.observability.svc:4317

Step 4 — Route whole traces to one replica. A gateway pool behind an ordinary load balancer splits a trace's spans across replicas, which silently defeats any trace-level policy: each replica decides on a fragment, and the result is traces that are present but missing most of their spans. The collector's own load-balancing exporter solves this by hashing the trace identifier, and it is the piece most often missing from a first gateway deployment.

# a thin routing tier in front of the sampling gateways
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { timeout: 5s, tls: { insecure: true } }
    resolver:
      k8s: { service: otel-gateway-sampling.observability }

Step 5 — Size each tier from its own arithmetic. An agent's queue needs to cover the time a gateway rollout takes, multiplied by one node's export rate. A gateway's queue needs to cover a backend outage multiplied by the whole fleet's rate. Those are different numbers by two or three orders of magnitude, and copying one tier's configuration to the other is the usual cause of a gateway that runs out of memory during the first backend incident.

Expected Output: both tiers reporting, with the backlog attributable to a tier rather than to the pipeline in general.

# agent, node-12
otelcol_receiver_accepted_spans 41221
otelcol_exporter_sent_spans     41221
otelcol_exporter_queue_size     8

# gateway, replica-2
otelcol_receiver_accepted_spans 984120
otelcol_exporter_sent_spans     981004
otelcol_exporter_queue_size     3116

What belongs at which tier

The placement rule that generalises is about the scope of the data a processor needs to do its job.

Per-record processors belong on the agent. Resource detection, attribute renaming, redaction of a field, dropping a span by name, batching: each of these looks at one record and decides. Running them close to the source is cheaper, because the data is smaller after filtering and the gateway then handles less of it, and it is more robust, because the decision does not depend on anything arriving.

Cross-record processors belong on the gateway. Tail sampling, trace-level metrics generation, deduplication, and anything that computes a rate. These need a window of data from many sources, and the gateway is the only tier that has one.

Destination concerns belong on the gateway too. Which backend, which credential, which copy goes where. Not because an agent cannot hold a token — it can — but because there are a hundred agents and three gateways, and a credential rotation across a hundred copies is a project rather than a change.

One nuance is worth flagging. Redaction is a per-record concern and therefore belongs on the agent by this rule, but it is also a correctness-critical one: if the agent's rule is wrong, sensitive data has already left the process and is on the node. For anything genuinely sensitive the right place is earlier still, in the application's own formatter, as described in redacting sensitive data in log records. The collector's redaction is a safety net, not the primary control.

A second nuance concerns cost attribution, which is not a technical property but decides more topologies than any of the above. An agent tier's cost appears as a per-node overhead in the platform's budget, where it is one line item that nobody argues about. A gateway tier's cost appears as a deployment somebody owns, sized against the whole fleet's traffic, and it grows when any team increases its telemetry volume. Teams that would never notice the agent will notice the gateway, and the conversation that follows is usually the first honest discussion a fleet has about how much telemetry it produces. That conversation is worth having, and it is easier when the numbers are per service — which is one more argument for the volume estimation described in estimating telemetry volume from a Python service.

Finally, the two tiers age differently. An agent's configuration is stable once it is right, because its job — receive locally, enrich, batch, forward — does not change when the business does. A gateway's configuration changes whenever a backend, a contract, a retention policy or a sampling budget changes, which in most organisations is several times a year. Treating the gateway's configuration as a versioned artefact with a review process, and the agent's as infrastructure that is rarely touched, matches the rate at which each actually moves.

Configuration options

Verification

The check that matters is whether a trace survives the gateway intact. Send one trace with a deliberate failure in a downstream service and count the spans that arrive.

# how many spans of a known trace reached the backend
TRACE=9f2a71c4f0b84c2e9d5f1a7b3c8e6d02
curl -s "http://backend/api/traces/${TRACE}" | python3 -c "
import json,sys
doc = json.load(sys.stdin)
print('spans stored:', len(doc['spans']))
print('services:', sorted({s['process']['serviceName'] for s in doc['spans']}))"

Expected Output: every span, from every service that participated.

spans stored: 5
services: ['api-gateway', 'checkout', 'inventory', 'payments']

A count lower than the number of services involved, on a trace that was kept, is the signature of the routing problem in step 4 — and it is worth testing explicitly, because nothing else reports it.

Common mistakes

A gateway with no agent beneath it. Error signature: application export failures during gateway rollouts. Root cause: every export crosses the network, so the gateway's availability is the application's availability. Remediation: add the per-node tier; the gateway then only ever talks to agents, which retry patiently.

Tail sampling behind a round-robin balancer. Error signature: traces present but missing spans from some services. Root cause: spans of one trace decided by different replicas. Remediation: route by trace identifier with a load-balancing exporter tier, as in step 4.

Copying the agent's configuration to the gateway. Error signature: the gateway killed for memory during the first backend outage. Root cause: a queue sized for one node's traffic receiving the fleet's. Remediation: size each tier from its own arithmetic, and set the memory limiter to refuse before the platform intervenes.

Credentials on every agent. Error signature: a credential rotation that takes a week and leaves stale tokens behind. Root cause: the backend token distributed to every node. Remediation: move the export to the gateway tier, and remove the secret from the agent's configuration entirely once the last one has moved.

Frequently Asked Questions