Python Observability

Estimating Telemetry Volume from a Python Service

Every telemetry cost conversation stalls at the same point: nobody knows what the service actually produces. This page covers getting that number — from one trace, one export payload and one request's worth of logs — and turning it into a monthly figure that survives contact with an invoice. It is a task article under telemetry cost and data volume control, part of the Python telemetry pipelines and delivery section.

One request, scaled up One request is decomposed into the telemetry it produces. It generates one server span, one span for the authentication call, eleven database spans from the ORM, two cache spans and one outbound HTTP span, sixteen in total at roughly one kilobyte each. It also produces four log records at about three hundred bytes each. It contributes to metric series but adds no new ones, because the series count is a property of the instrumentation rather than of traffic. Multiplying the per-request figures by the service's request rate and by the seconds in a month gives the monthly volume, and applying the compression ratio gives the figure the backend is likely to bill. The diagram marks the database spans as the largest single contributor, which is typical and is where a reduction has the most effect. one request in a typical instrumented service spans server auth 11 database spans — the bulk of it cache ×2 http 16 spans × ~1 KiB = 16 KiB raw per request logs 4 records × ~300 B = 1.2 KiB — and no sampling term metrics adds no series at all series count comes from the instrumentation, not from traffic 17.2 KiB per request × 420 requests/s × 2.6 million seconds = 18.7 TB raw per month compressed at roughly eight to one, that is about 2.3 TB — check which figure the contract bills the eleven database spans are 69% of the bytes, which is where a reduction actually helps
The multiplication is trivial. Getting the two per-request numbers right is the entire exercise, and both come from a single real request.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-http>=1.27.0,<2.0.0"

Implementation

Step 1 — Count spans per request from one complete trace. Open a single trace in whatever backend you have and count its spans, excluding the spans belonging to downstream services if you are estimating this service only. This number is the one people guess at most wildly, and the guess is nearly always low, because automatic instrumentation of an ORM produces a span per statement and a request that looks like one query frequently issues a dozen. If the number surprises you, that is itself a finding, and detecting repeated query patterns with traces is the next page to read.

Step 2 — Measure bytes per span from a real export. Point the exporter at a local file-writing endpoint for a minute, or capture one request body, and divide. Estimating this from the span's fields underestimates it substantially, because attributes dominate: a database span carrying a statement, a connection string and a set of semantic convention attributes is several times the size of a bare one.

# capture.py — write one export payload to disk for measurement
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Point at a local endpoint that logs bodies; any HTTP echo server works.
provider.add_span_processor(SimpleSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:8080/v1/traces")))

# measure.py
import gzip, json, pathlib

raw = pathlib.Path("export-body.json").read_bytes()
doc = json.loads(raw)
spans = sum(len(ss["spans"]) for rs in doc["resourceSpans"] for ss in rs["scopeSpans"])

print(f"spans            {spans}")
print(f"bytes per span   {len(raw) / spans:8.0f} raw")
print(f"                 {len(gzip.compress(raw)) / spans:8.0f} compressed")

Expected Output:

spans            512
bytes per span        998 raw
                      129 compressed

Step 3 — Count log records per request. Take one request's log output at the level production actually runs at, and count. Logs deserve separate attention because they have no sampling term: every record produced is a record stored, so the per-request count multiplies directly into the bill. Four records per request at a thousand requests per second is three hundred and forty million records a month, which is a number worth seeing before somebody adds a fifth.

Step 4 — Count active series rather than instruments. A service with twelve metric definitions can have twelve series or twelve thousand, depending entirely on labels. The calculation is per instrument: multiply the cardinality of each of its labels together, then sum across instruments. Histograms multiply again by their bucket count, which is why a latency histogram with a generous ladder is usually the largest single contributor.

# series.py — the calculation that matters, done explicitly
INSTRUMENTS = {
    "http_server_duration": {"labels": {"route": 24, "method": 4, "status": 6}, "buckets": 12},
    "http_server_requests": {"labels": {"route": 24, "method": 4, "status": 6}, "buckets": 1},
    "db_pool_in_use":       {"labels": {"pool": 3}, "buckets": 1},
}

total = 0
for name, spec in INSTRUMENTS.items():
    combos = 1
    for cardinality in spec["labels"].values():
        combos *= cardinality
    series = combos * spec["buckets"]
    total += series
    print(f"{name:26s} {combos:6d} combinations × {spec['buckets']:2d} = {series:7d} series")
print(f"{'total':26s} {total:24d} series")

Expected Output:

http_server_duration          576 combinations × 12 =    6912 series
http_server_requests          576 combinations ×  1 =     576 series
db_pool_in_use                  3 combinations ×  1 =       3 series
total                                             7491 series

Step 5 — Multiply, and apply the right compression assumption. The monthly figure is per-request bytes times request rate times seconds in a month. Whether to use the raw or compressed number depends entirely on the contract, and the two differ by roughly a factor of eight for telemetry payloads, which is more than enough to make an estimate useless if the wrong one is chosen.

Step 6 — Record the assumptions alongside the number. An estimate whose inputs are not written down cannot be updated when one of them changes, and one of them always changes: a new endpoint adds routes, a library upgrade adds spans, a release adds a log statement. Keeping the inputs in a small file next to the service means the next estimate takes two minutes rather than starting over.

Where a span's kilobyte goes A single span's wire size is broken down. Its intrinsic fields — trace identifier, span identifier, parent, name, kind and the two timestamps — account for roughly one hundred and twenty bytes. Its own attributes account for around six hundred bytes, dominated by long string values such as a database statement or a full URL. Span events and links, when present, add perhaps a hundred more. The resource attributes describing the service and host are another two hundred bytes, but they are serialised once per export batch rather than per span, so across a batch of five hundred spans they contribute well under a byte each. The conclusion drawn is that attribute values are almost the entire cost, which means trimming one long attribute across the fleet does more than any change to how spans are structured. one span, about a kilobyte on the wire ids + times attribute values — statements, URLs, headers events resource ~120 B ~600 B — this is the whole cost ~100 B ~200 B the resource is serialised once per batch, not once per span across a batch of 500 spans it costs under a byte each — which is why resource attributes are effectively free trimming one long attribute fleet-wide beats any structural change to how spans are produced
Attribute values are the span. Everything else, including the resource that describes the whole service, rounds to nothing by comparison.

What the estimate is for

An estimate that is only used to predict a bill is doing a fraction of the work it could. Three other uses are more valuable.

Sizing the pipeline. Every buffer decision in buffering telemetry during backend outages starts from the production rate, and the same numbers produce it. A fleet that knows its per-request figures can size queues, collectors and network capacity from arithmetic rather than by observing failures.

Comparing services. Bytes per request is comparable across services in a way that totals are not. A service producing ten times its neighbour's per-request volume is either doing something genuinely more complex or has a problem, and the ratio is what prompts the question. In practice the answer is usually an ORM issuing far more statements than the developer believes, which the span count exposes immediately.

Predicting the effect of a change. Before adopting automatic instrumentation for a new library, before adding a label to a metric, before adding a log statement to a hot path — the estimate says what it will cost. A label with fifty values multiplies a metric's series by fifty; seeing that written down changes the decision more reliably than any guideline.

There is also a defensive use. A service whose volume is understood can respond to a cost reduction request with a specific proposal — drop the eleven database spans to one summary span, saving sixty-nine percent — rather than with an unfocused reduction that costs every future investigation. The estimate is what converts a mandate into an engineering decision.

The numbers that are hardest to estimate

Two inputs resist estimation and are worth measuring rather than reasoning about.

The first is log volume under failure. A service producing four records per request in normal operation can produce forty during an incident, because every failure path logs, retries log, and the error handlers that are dormant for months all run at once. The volume peak therefore coincides exactly with the moment the pipeline is under most stress and with the moment the data matters most. Estimating from steady-state traffic understates the peak by an order of magnitude, and a buffer sized on the steady state will not survive the event it exists for. The practical approach is to take the worst hour from the last quarter and use that as the design figure for logs, while using the average for traces.

The second is the effect of automatic instrumentation on a library you have not used before. Adding an instrumentation package for a message queue, a cache or a new database driver can add anywhere between one span per operation and one span per network round trip, and the difference is not documented in a way that can be predicted. Enabling it in staging under representative load for an hour and re-running the span count from step 1 costs almost nothing and answers the question exactly. Doing that before the fleet-wide rollout, rather than after, is the difference between a planned increase and a surprise.

One service, worked through A worked estimate of daily telemetry volume for one Python service handling 200 requests per second. Spans: 200 requests times 8 spans times 1 kilobyte, sampled at 10 percent, is 160 kilobytes per second, about 14 gigabytes per day. Logs: 200 requests times 3 records times 0.5 kilobytes is 300 kilobytes per second, about 26 gigabytes per day. Metrics: 5000 active series, each sample about 2 bytes compressed, scraped every 15 seconds, is under 1 gigabyte per day. The total is about 41 gigabytes per day, and logs are the largest part. The note says multiplying rate by items per request by size per item is enough to find which signal to look at first. signal arithmetic per day spans 200/s × 8 × 1 KB × 10% sampled ~14 GB logs 200/s × 3 records × 0.5 KB ~26 GB metrics 5 000 series × 15 s scrape <1 GB total logs are the largest share here ~41 GB rate × items per request × bytes per item — enough to know where to look first
A three-factor multiplication per signal gets within a factor of two, which is enough to pick the first target.

Configuration options

Input How to get it Typical value Sensitivity
Spans per request count one real trace 3–40 very high
Bytes per span measure one export 800–1 400 raw moderate
Log records per request one request at production level 2–8 very high
Bytes per record measure one line 200–600 moderate
Active series multiply label cardinalities 500–50 000 very high
Compression ratio gzip a real payload 6–10× high
Retention the contract 7–30 days high

Verification

Check the estimate against reality after a week, because the point of an estimate is to be corrected.

# what the collector actually forwarded, over a day
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_sent_(spans|log_records)'

# against the estimate
python3 -c "
rps, spans_per_req = 420, 16
print(f'predicted spans/day: {rps * spans_per_req * 86400:,}')"

Expected Output: a prediction within a factor of two, which is close enough to act on.

otelcol_exporter_sent_spans 574_291_002
predicted spans/day: 580,608,000

A prediction that is off by an order of magnitude usually means the span count per request was taken from a simple endpoint rather than a representative one, which is worth correcting before the number is used for anything else.

Common mistakes

Counting spans from the simplest endpoint. Error signature: an estimate several times below reality. Root cause: a health check or a static route used as the sample. Remediation: take the trace of a representative business operation, ideally the one with the highest traffic.

Estimating span size from the schema. Error signature: an estimate half the real figure. Root cause: attributes, not fields, dominate the size. Remediation: measure a real payload.

Ignoring histogram buckets in the series count. Error signature: a metrics estimate an order of magnitude low. Root cause: counting one series per label combination when a histogram produces one per bucket. Remediation: multiply by the bucket count, as in step 4.

Using the wrong compression assumption. Error signature: an estimate eight times the invoice, or an eighth of it. Root cause: raw versus compressed. Remediation: check the contract and measure both.

Estimating once. Error signature: a figure that was accurate a year ago. Root cause: the inputs change with every release. Remediation: keep the inputs in a file and re-run the arithmetic when something notable changes.

Frequently Asked Questions

How many spans does a typical Python request produce?

It varies by an order of magnitude. A minimally instrumented service produces two or three — one server span and one client span. A service with automatic instrumentation for the web framework, the database driver, the HTTP client and the cache routinely produces fifteen to forty, most of them from the database layer.

How large is a span on the wire?

Around one kilobyte raw for a typical instrumented span, dominated by attributes rather than by the span's own fields. Resource attributes are shared across all spans in an export, so they cost far less per span than their size suggests. Compression usually removes seventy to ninety percent.

Do I need to estimate before instrumenting?

For a fleet, yes — the difference between three spans per request and thirty is the difference between a routine bill and a project. For one service, instrument first and measure after a week, because a real measurement takes ten minutes and a good estimate takes longer.

Why is my estimate so far from the invoice?

Usually compression, sometimes retention. A backend billing compressed ingest will charge roughly an eighth of the raw figure, and one billing on stored volume after replication may charge several times it. Check which the contract uses before reconciling.

Related Guides