Log Retention and Tiering Strategy

Log storage costs are dominated by data nobody reads. The queries land in the last few hours; the invoice covers the last few months. This page covers measuring where the queries actually go, splitting records by why they are kept, and setting tier boundaries that follow the evidence. It is a task article under telemetry cost and data volume control, part of the Python telemetry pipelines and delivery section.

Prerequisites

Nothing to install. What is needed is access to the store's query log and to the volume figures per index or stream.

# the two inputs: query ages, and volume per day
curl -s 'http://logs:9200/_cat/indices/logs-*?v&s=index&h=index,docs.count,store.size'

Implementation

Step 1 — Measure the age of the data each query touched. Most stores log the time range of every query. Extracting the age of the oldest point each query reached, and plotting the distribution, replaces a policy argument with a measurement. The result is nearly always more skewed than anybody expects, and it is the single most persuasive artefact available when proposing a retention change.

# query_ages.py — how far back queries actually reach
import collections, datetime, json, pathlib

buckets = collections.Counter()
for line in pathlib.Path("query-log.ndjson").read_text().splitlines():
    q = json.loads(line)
    age_h = (datetime.datetime.now().timestamp() - q["range_start_epoch"]) / 3600
    if age_h <= 24:        buckets["< 1 day"] += 1
    elif age_h <= 168:     buckets["1–7 days"] += 1
    elif age_h <= 720:     buckets["1–4 weeks"] += 1
    else:                  buckets["> 1 month"] += 1

total = sum(buckets.values())
for label in ("< 1 day", "1–7 days", "1–4 weeks", "> 1 month"):
    print(f"{label:12s} {buckets[label]:7d}  {buckets[label]/total:6.1%}")

Expected Output:

< 1 day        184203   91.4%
1–7 days        14820    7.4%
1–4 weeks        2010    1.0%
> 1 month         421    0.2%

Step 2 — Split records by why they are retained. This is the step that makes everything else possible. Operational logs are kept because somebody might debug with them, and their value falls away with the query distribution above. Audit and security records are kept because a policy or a regulator says so, and their value is constant until the retention period ends. Mixing them forces the operational data onto the audit retention, which is where most of the over-retention in a typical fleet comes from.

# route by a field the application sets, not by guesswork
connectors:
  routing/by_purpose:
    default_pipelines: [logs/operational]
    table:
      - context: log
        condition: 'attributes["record.class"] == "audit"'
        pipelines: [logs/audit]
      - context: log
        condition: 'severity_number >= SEVERITY_NUMBER_ERROR'
        pipelines: [logs/errors]

Step 3 — Size the hot tier from the query distribution, with headroom. The measurement above suggests a boundary; the boundary should sit somewhat beyond it, because the cost of being slightly too generous is small and the cost of an incident review finding nothing is not. A distribution like the one above justifies seven days of indexed storage comfortably, where thirty was probably in place.

Step 4 — Archive with enough structure to be searchable. Data moved to object storage should remain queryable, slowly. Writing it partitioned by day and service, in a columnar format, means an audit or an incident review can still answer a specific question in minutes without a restore. Writing it as opaque compressed blobs means it is retained rather than kept.

exporters:
  awss3:
    s3uploader:
      s3_bucket: telemetry-archive
      s3_prefix: logs/service={{.service}}/date={{.date}}
      file_prefix: part-
    marshaler: otlp_json

Step 5 — Keep error records longer than everything else. Errors are a tiny fraction of volume and a large fraction of value, which makes them the best possible candidate for a separate, longer retention. A fleet keeping all logs for seven days and error-level records for ninety pays almost nothing for the second policy and gains the ability to answer "has this happened before" — which is the question most likely to reach past a week.

Step 6 — Automate the deletion and write down who agreed to it. Retention that depends on somebody remembering is unbounded retention. A lifecycle policy with the boundary in it, and a note recording who approved that boundary and when, converts an ongoing cost into a decision that can be revisited deliberately.

What retention is really deciding

Underneath the cost arithmetic, a retention policy is a statement about which questions the organisation intends to be able to answer, and it is worth making that statement explicitly rather than arriving at it by way of a storage budget.

Debugging a live problem needs the last few hours at full fidelity, searchable without knowing in advance what to look for. This is the only tier that has to be fast and the only one that has to be complete, and it is short.

Understanding a recurring problem needs error records across weeks, so that "this has happened twice before, both times after a deploy" is answerable. This does not need the debug output from those occasions, only the errors, which is why splitting by severity is so effective.

Reconstructing an incident weeks later needs whatever the review asks for, which is unpredictable in content and highly predictable in shape: a specific service, a specific window, already known. Slow archived storage serves this perfectly well.

Satisfying an obligation needs whatever the obligation specifies, kept in a way that can be produced on request, with its integrity defensible. This is a different problem from the other three, and mixing it with them is what produces both over-retention of operational data and under-protection of the records that matter.

A fleet that writes these four purposes down usually discovers that its current policy serves the first and fourth accidentally and the middle two not at all. Reorganising around them tends to reduce cost and improve the answers simultaneously, which is unusual enough to be worth the afternoon it takes.

Getting the change agreed

Shortening retention is a technical change with an organisational blocker, and the blocker is almost always the same: nobody wants to be the person who deleted the data that would have answered the question. Three things make the conversation tractable.

The first is the query age distribution from step 1. It converts an opinion into a measurement, and it is remarkably persuasive precisely because everybody's intuition about it is wrong. Presenting it alongside the proposed boundary, with the tail visible, pre-empts the objection rather than arguing with it.

The second is that tiering is not deletion. The proposal is not "we will keep less"; it is "we will keep the same data, indexed for a week and archived thereafter". The only thing lost is the ability to run an exploratory search against last month, which the distribution shows nobody does. Framing it this way moves the discussion from risk to cost, where it belongs.

The third is a stated exception process. A team that genuinely needs longer indexed retention for a specific service can have it, at a cost that is now visible and attributable. Offering this removes the incentive to argue for a longer default on everyone's behalf, and in practice it is rarely taken up — which is itself informative about how much the longer retention was worth.

Configuration options

Verification

Check that the archive is actually queryable, because an archive that has never been read is a backup nobody has restored.

# find one known record in the archive, by service and day
aws s3 ls s3://telemetry-archive/logs/service=checkout/date=2026-06-14/ | head -3

# and confirm it can be searched without a restore
duckdb -c "SELECT count(*) FROM read_json_auto(
  's3://telemetry-archive/logs/service=checkout/date=2026-06-14/*.json')
  WHERE severity_text = 'ERROR'"

Expected Output: files present and a query returning in seconds to minutes.

part-000012.json
part-000013.json
┌──────────────┐
│ count_star() │
│        1 284 │
└──────────────┘

A query that cannot run without a restore step means the archive format or the partitioning needs revisiting before the hot tier is shortened, not after.

Common mistakes

One retention for everything. Error signature: a large bill and a policy nobody can justify. Root cause: audit requirements applied to debug output. Remediation: split by record purpose at the collector, as in step 2.

Setting the boundary from a round number. Error signature: thirty days of indexed storage serving queries that almost all land inside one day. Root cause: no measurement. Remediation: plot the query age distribution and set the boundary beyond its tail.

Archiving into an unqueryable format. Error signature: an incident review that cannot use data the organisation is paying to keep. Root cause: opaque compressed blobs with no partitioning. Remediation: partition by service and day, store in a format a query engine can read directly.

Keeping everything longer instead of errors longer. Error signature: cost proportional to debug volume, with no improvement in answering historical questions. Root cause: severity not used as a retention dimension. Remediation: route error records to a longer-retention destination; they are a small fraction of volume.

No automated deletion. Error signature: storage that grows without bound and a policy that exists only in a document. Root cause: manual cleanup. Remediation: lifecycle policies with the boundary encoded, and an alert if they stop running.

Frequently Asked Questions