Python Observability

Routing Telemetry to Multiple Backends

Every fleet eventually needs the same telemetry in more than one place — a vendor during a migration, an archive for retention, a regulated tenant's own store, or a cheap copy of everything alongside an expensive copy of a subset. Doing this in application code means every service redeploys whenever the answer changes. This page covers doing it in the collector, where it is a configuration change. It is a task article under collector topology and deployment, part of the Python telemetry pipelines and delivery section.

One input, three destinations, three volumes A single OTLP receiver accepts everything a fleet produces. Three pipelines branch from it. The first applies no filter and exports to cheap object storage, so it carries one hundred percent of the data at the lowest price per byte. The second applies a tail sampling policy that keeps errors, slow traces and a small probabilistic sample, and exports to the interactive vendor backend that engineers query during incidents, carrying roughly eight percent of the volume at a much higher price per byte. The third applies a routing rule that selects only records whose tenant attribute names a regulated customer, and exports those to that customer's own store, carrying about two percent. A fourth arrow leaves the routing rule for records whose tenant attribute is missing or unrecognised and goes to a quarantine destination rather than to any tenant, which is the branch that prevents a data incident. one receiver, three destinations, three prices receiver 100% no filter 100% · cheapest bytes object storage archive tail sampling ~8% · dear bytes interactive vendor route by tenant ~2% · one customer regulated tenant store quarantine — no tenant match
The unmatched branch is the one that matters. Without it, a record missing its tenant attribute goes wherever the last rule happens to send it.

Prerequisites

The application needs no changes beyond setting the resource attributes the routing rules will key on.

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

export OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,deployment.environment=prod,tenant.id=acme"

Implementation

Step 1 — Fan out by listing several exporters. The simplest multi-destination arrangement needs no new component: a pipeline with two exporters delivers every record to both. Each exporter has its own queue and its own retry state, so a failure at one destination does not affect the other. This is the right shape when every record belongs everywhere, which is the case for an archive copy alongside a working copy.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/vendor, otlphttp/archive]     # both get everything

Step 2 — Give each destination its own pipeline when they need different data. Fan-out sends identical data everywhere, which is wasteful when one destination costs ten times the other. Separate pipelines, fed by the same receiver, let each destination have its own processors — a sampler on the expensive path, nothing on the cheap one.

processors:
  tail_sampling/interactive:
    decision_wait: 10s
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency,     latency: { threshold_ms: 500 } }
      - { name: sample, type: probabilistic, probabilistic: { sampling_percentage: 5 } }

service:
  pipelines:
    traces/archive:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/archive]
    traces/interactive:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling/interactive, batch]
      exporters: [otlp/vendor]

Step 3 — Select a destination with a routing connector. When records belong in different places rather than all places, a routing connector reads an attribute and directs each record to the pipeline that matches. The value it keys on must be a resource attribute rather than a span attribute, because resource attributes are stable across every record from one process and therefore cannot be missing from some spans of a trace and present on others.

connectors:
  routing:
    default_pipelines: [traces/quarantine]      # the important line
    table:
      - context: resource
        condition: 'attributes["tenant.id"] == "acme-regulated"'
        pipelines: [traces/tenant_acme]
      - context: resource
        condition: 'attributes["deployment.environment"] == "prod"'
        pipelines: [traces/interactive, traces/archive]

Step 4 — Make the default route harmless and visible. A routing table without a default sends unmatched records somewhere the implementation chooses, and "somewhere" is occasionally another tenant's destination. A quarantine pipeline — a cheap store nobody queries during incidents, with an alert on its record rate — turns a potential data incident into a monitoring signal. A non-zero quarantine rate means some service is not setting the attribute, which is a bug worth fixing and not worth a breach report.

Step 5 — Filter what each destination does not need. A destination billed per byte should not receive attributes nobody queries. A per-pipeline filter or attribute processor removes them after the routing decision, so the same record can be complete in the archive and trimmed in the expensive store.

processors:
  attributes/trim_for_vendor:
    actions:
      - { key: http.request.header.cookie, action: delete }
      - { key: db.statement, action: delete }        # kept in the archive copy

Expected Output: per-exporter counters showing three destinations with three volumes, and an empty quarantine.

otelcol_exporter_sent_spans{exporter="otlphttp/archive"}     984120
otelcol_exporter_sent_spans{exporter="otlp/vendor"}           78106
otelcol_exporter_sent_spans{exporter="otlp/tenant_acme"}      19842
otelcol_exporter_sent_spans{exporter="otlphttp/quarantine"}       0
Why per-exporter queues matter Three exporters are drawn side by side during a two minute outage of one backend. The archive exporter continues sending normally, its queue near empty. The vendor exporter cannot reach its destination, so its own sending queue fills steadily over the outage and begins dropping once full, while its retry timer backs off. The tenant exporter is unaffected and continues. The key property illustrated is that the queues are independent: the failing destination consumes only its own buffer, so the healthy destinations continue at full rate and lose nothing. The contrast drawn underneath is with a single shared queue, where the failing destination would hold records that the healthy ones were waiting to send, converting one backend's outage into a total pipeline stall. one backend down, two minutes archive sending normally · queue near empty throughout vendor ok own queue filling, backing off dropping its own records only tenant unaffected independent queues are what make fan-out safe with one shared queue, the slowest destination sets the rate for all of them and an outage stalls everything
Each exporter buffers and retries alone. That isolation is the reason to fan out inside one collector rather than by running several.

Getting the routing key right

Almost every routing failure traces back to the attribute the rule keys on, and three properties separate a key that works from one that causes an incident.

It must be a resource attribute, not a span or record attribute. Resource attributes describe the process that produced the telemetry and are identical on every record it emits. A span attribute can differ between spans of one trace, which means a routing rule keyed on it can send half a trace to one destination and half to another — producing two incomplete traces in two stores, neither of which says it is incomplete. This is the same class of problem as sampling a trace across replicas, and it is equally invisible afterwards.

It must be set at the source, not inferred. A rule that derives a tenant from a namespace name, a service name prefix or an image tag works until somebody renames something. Setting the attribute explicitly in the application's resource, as in the environment variable above, means the routing decision depends on a value the service owner chose deliberately. Configuring resource attributes for Python services covers how to attach it once so every signal carries it.

It must have a safe behaviour when absent. Every rule set needs an answer to "what if this attribute is missing", and the only acceptable answer when tenancy is involved is a destination that belongs to nobody. A missing attribute is not rare: it happens on a service's first deploy, on a job that was written before the convention existed, and on anything emitted by a library rather than by application code. Sending those to quarantine means the failure is a dashboard line rather than a disclosure.

A fourth consideration is worth adding for migrations specifically. When moving between vendors, the natural arrangement is to send everything to both for a period, compare, and then remove the old one. The comparison is the part that is usually skipped and the part that catches the problems: different backends disagree about span counts when one of them applies its own ingest-side sampling, and discovering that after the cutover is considerably worse than discovering it during the overlap.

What it costs to run

Multi-destination routing is cheap to configure and not free to operate, and three ongoing costs are worth budgeting for before adopting it.

The first is memory in the gateway, which scales with destinations rather than with inputs. Each exporter holds a sending queue sized for the outage it should survive, so three destinations mean three queues; adding a fourth backend is a capacity change, not just a configuration change.

The second is alerting surface. Each destination fails independently, which is the property that makes fan-out safe, and it also means three sets of failure counters to watch. A dashboard that aggregates across exporters hides exactly the case the isolation was designed for — one destination down while the others are healthy — so the alerts have to be per exporter, with a route to whoever owns that destination.

The third is the drift between what each backend holds. Once the copies differ, a question answered from one may not be answerable from another, and engineers do not carry a mental model of which destination holds what. Writing down, in one place, what each destination receives and for how long is a small piece of documentation that saves a recurring argument during incidents. The rule that keeps it manageable is that the differences should be few and large — full versus sampled, full attributes versus trimmed — rather than many and subtle.

Common routing rules A table of routing rules seen in practice and what each achieves. Route by signal: traces to the tracing backend, metrics to Prometheus remote write, logs to the log store. Route by tenant attribute: each customer's telemetry to its own backend or index, for isolation or billing. Route by environment: production to the paid backend, staging to a cheaper one with short retention. Route security and audit logs to a separate store with restricted access. Copy a sample of traces to a second vendor during an evaluation. The note says the routing key must be an attribute present on every record, which usually means a resource attribute set by the SDK or added by the agent. route by to achieve signal each signal to its own backend tenant attribute isolation or per-customer billing environment staging to cheaper, shorter storage log category audit and security to a restricted store sampled copy a second vendor during evaluation the routing key must be on every record — usually a resource attribute
Routing splits one stream by a key. The rule only works if every record carries that key.

Configuration options

Concern Mechanism Note
Same data everywhere several exporters in one pipeline simplest; independent queues
Different data per destination separate pipelines, same receiver filters differ per path
Different destination per record routing connector key on a resource attribute
Unmatched records default_pipelines must be quarantine, never a tenant
Per-destination trimming attribute processor after routing archive keeps what the vendor drops
Failure isolation per-exporter sending queue one outage stays local
Cost visibility per-exporter sent counters volume per destination, per service

Verification

The check worth running is the negative one: prove that a record with no tenant attribute lands in quarantine rather than anywhere else.

# emit one span deliberately missing the routing attribute
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider(resource=Resource.create({"service.name": "routing-probe"}))
trace.set_tracer_provider(provider)
with trace.get_tracer(__name__).start_as_current_span("no-tenant-attribute"):
    pass
provider.shutdown()

Expected Output: the quarantine exporter's counter moves and no tenant destination does.

otelcol_exporter_sent_spans{exporter="otlphttp/quarantine"}   1
otelcol_exporter_sent_spans{exporter="otlp/tenant_acme"}      0

Common mistakes

No default route. Error signature: records from a new service appearing in an unrelated tenant's store. Root cause: an unmatched record falling through to whichever pipeline the configuration listed last. Remediation: set default_pipelines to a quarantine destination and alert on its rate.

Routing on a span attribute. Error signature: traces split across two backends, each copy missing spans. Root cause: the key differs between spans of one trace. Remediation: move the attribute to the resource so it is constant per process.

Sizing the gateway for its input. Error signature: memory pressure that appeared when a second destination was added. Root cause: each exporter has its own queue, so memory scales with destinations rather than with receivers. Remediation: size for the sum of the destinations and raise the memory limiter accordingly.

Trimming before routing. Error signature: the archive copy missing the attributes it exists to preserve. Root cause: an attribute processor placed in the shared part of the pipeline rather than in the per-destination branch. Remediation: put per-destination processors after the split, so the cheap copy keeps what the expensive one drops.

Assuming both backends agree. Error signature: a migration cutover where trace counts differ by several percent. Root cause: ingest-side sampling or rejection at one vendor. Remediation: compare counts during the overlap period, per service, before removing the old destination.

Frequently Asked Questions

Does sending to two backends double the collector's memory?

It doubles the queueing, because each exporter has its own sending queue, and it roughly doubles the outbound bandwidth. It does not double the receiving or processing cost, which is shared. Size the gateway for the sum of its destinations rather than for its inputs.

What happens when one backend is down and the other is fine?

Each exporter retries independently, so the healthy destination is unaffected. The failing one fills its own queue and then drops. This isolation is the main reason to fan out in the collector rather than by running two collectors.

How do I route different tenants to different stores?

Put the tenant identifier in a resource attribute at the source, then use a routing connector keyed on it. The critical detail is the default route: anything without a recognised tenant must go to a quarantine destination, never to a tenant's store.

Can I send a sampled copy to one backend and everything to another?

Yes, and it is one of the most useful arrangements available. Give each destination its own pipeline: full fidelity to cheap archival storage, a sampled and filtered subset to the expensive interactive backend.

Related Guides