Python Observability

Graceful Shutdown and Telemetry Flush

The spans and log records produced in a process's final seconds describe why it stopped, which makes them the most valuable telemetry it ever produces and the most likely to be lost. This page covers the handler, the ordering and the grace period that get them out, for long-running servers, prefork workers and processes whose whole life is shorter than one export interval. It is a task article under backpressure, retries and delivery guarantees, part of the Python telemetry pipelines and delivery section.

What the last ten seconds contain Two termination sequences. In the upper one the process receives a termination signal and exits without a handler. The batch processor last exported five seconds earlier, so everything produced since then is in its queue, including the log records about draining, the spans of the requests that were in flight and the error from the one request that failed during shutdown. All of it is discarded with the process, and nothing anywhere records that it happened. In the lower sequence a signal handler runs: the server stops accepting new connections, in-flight requests complete, and only then are the tracer, meter and logger providers shut down, which flushes each of their queues. The same records leave the process, and a final line confirms the flush completed. The window is marked as typically under a second against a local collector, against a grace period of thirty seconds. SIGTERM, then ten seconds no handler last export queued: drain logs, in-flight spans, one error exit all of it discarded handler, in the right order stop accepting in-flight requests finish flush exit everything delivered the flush is typically under a second against a local collector — the drain is what needs the grace period ordering matters: flushing before the drain loses the telemetry the drain produces
The ordering is the whole trick. Flushing first and draining afterwards produces a clean shutdown with no record of it.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

Implementation

Step 1 — Install a handler for the signal your platform actually sends. Container platforms send a termination signal and then wait; a process with no handler receives the default disposition, which ends it immediately. Installing a handler is what buys the window. Note that the handler runs on the main thread between bytecode instructions, so it must not block indefinitely — which is why every call inside it needs a timeout.

import signal
import threading

_shutting_down = threading.Event()

def _on_term(signum, frame):
    _shutting_down.set()          # cooperative: the server loop checks this

signal.signal(signal.SIGTERM, _on_term)
signal.signal(signal.SIGINT, _on_term)

Step 2 — Stop accepting work before flushing. This ordering is the part that is most often wrong, and the failure is subtle: a service that flushes immediately on the signal, then drains, has shut down its providers before the drain produces its telemetry. The result is a clean shutdown with no record of it, which is worse than no handler at all because it looks correct. Stop accepting, let in-flight work finish, then flush.

# FastAPI / Starlette lifespan — the ordering made explicit
from contextlib import asynccontextmanager
from opentelemetry import trace, metrics, _logs

@asynccontextmanager
async def lifespan(app):
    yield
    # 1. Reached only after the server has stopped accepting and drained.
    # 2. Now the last spans and log records exist, so flush them.
    trace.get_tracer_provider().shutdown()
    metrics.get_meter_provider().shutdown()
    _logs.get_logger_provider().shutdown()

Step 3 — Shut down every provider. Each signal has its own provider, its own queue and its own export thread, and shutting down the tracer provider does nothing for the metrics or logs queues. A service that flushes only traces loses the metric points and the log records from the same window, which in practice are the ones an engineer looks at first.

Step 4 — Give the grace period room for both phases. The platform kills the process when the grace period expires, wherever it has got to. That period has to cover the longest in-flight request plus the flush. Against a local collector the flush is usually well under a second, so the request drain dominates and the number should come from the service's own latency distribution rather than from a default.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            # 1. Let the load balancer notice before the server stops accepting.
            command: ["sh", "-c", "sleep 5"]

Step 5 — Make it idempotent and make it visible. A shutdown path can be entered twice — a signal during an already-running shutdown, or a handler plus an atexit hook — and shutting a provider down twice should be harmless rather than an exception in the final seconds. Emitting a line when the flush completes is equally important: it is the only evidence the flush happened, and its absence in a terminated pod's log is the fastest possible diagnosis.

import logging
import threading

log = logging.getLogger(__name__)
_flushed = threading.Event()

def flush_telemetry(reason: str = "signal") -> None:
    if _flushed.is_set():
        return                                  # idempotent
    _flushed.set()
    for name, provider in (
        ("tracer", trace.get_tracer_provider()),
        ("meter", metrics.get_meter_provider()),
        ("logger", _logs.get_logger_provider()),
    ):
        shutdown = getattr(provider, "shutdown", None)
        if callable(shutdown):
            try:
                shutdown()
            except Exception:                    # never block the exit path
                log.exception("provider shutdown failed", extra={"provider": name})
    log.info("telemetry flushed", extra={"reason": reason})

Expected Output: a terminating pod whose final line proves the window was used.

2026-09-18T14:12:31Z INFO  preStop: sleeping 5s for endpoint removal
2026-09-18T14:12:36Z INFO  server stopped accepting; 3 requests in flight
2026-09-18T14:12:38Z INFO  drain complete
2026-09-18T14:12:38Z INFO  telemetry flushed reason=lifespan
How much is in the queue when each kind of process ends Four process shapes are compared by the proportion of their telemetry that is unexported at the moment they stop. A long-running server exports every few seconds for hours, so only the final interval is at risk, which is a tiny fraction of its total output but contains everything about the shutdown. A prefork worker is the same, multiplied by the worker count, because every worker has its own queue and all of them terminate together during a deploy. A short-lived job that runs for forty seconds exports a handful of times, so the final interval is a meaningful fraction of the whole. A job that runs for three seconds never reaches its first scheduled export at all, so without an explicit flush one hundred percent of its telemetry is lost, every time, silently. The conclusion drawn is that the shorter the process, the more structural the flush requirement becomes. proportion of telemetry unexported at exit long-running server a few percent — but it is all of the shutdown prefork workers the same, once per worker, all at the same moment 40-second job a quarter of everything it produced 3-second job all of it — the first scheduled export never happens
For a server the flush recovers the interesting fraction. For a short job it is the difference between all the telemetry and none of it.

The cases that need more than a signal handler

The pattern above covers a long-running server. Three other shapes need something different.

Prefork workers. Under a prefork server each worker is a separate process with its own providers and its own queue, so the handler has to be installed in each worker rather than in the master. The master's own handler typically forwards the signal and then waits, which means the workers get their window — but only if the master's shutdown timeout is longer than the workers' flush. Configuring the two independently, with the master's larger, is the arrangement that works. The related provider-initialisation ordering is covered in Prometheus multiprocess mode with Gunicorn, and the same after-fork rule applies to the tracer provider.

Short-lived jobs. A batch task, a cron job or a data pipeline step may run for less time than one export interval, in which case no scheduled export ever happens and the flush is not a safety net but the only delivery mechanism. The clean pattern is a context manager around the whole job so the flush cannot be forgotten, and for very small jobs a simple span processor that exports on span end is a reasonable alternative — the volume is low enough that synchronous export costs nothing meaningful. This is developed further in logging from cron and batch jobs.

Processes killed without warning. An out-of-memory kill, a node failure or a forced termination runs no handler at all. Nothing in the process can help, which means the only protection is not keeping much in the process: a short export interval and a small queue lose less when the process disappears. This is the one case where a small queue is better than a large one, and it is worth weighing against the buffering arithmetic, since the two pull in opposite directions.

A fourth case deserves a mention because it produces the most confusing symptom: a process that exits normally after its work completes, without any signal. The interpreter runs atexit handlers here, so registering the flush there covers it — but atexit does not run on a signal, and a signal handler does not run on a normal exit. Both paths are needed, which is why the flush function above is written to be idempotent.

A shutdown that keeps the last records Five steps in a shutdown that does not lose telemetry. The process receives SIGTERM. It stops accepting new work, for example by failing its readiness check. It finishes in-flight requests. It calls shutdown on the tracer, meter and logger providers, which flush their queues to the exporter. It exits, before the platform's grace period ends and SIGKILL arrives. The note says the flush must fit inside the grace period, so its timeout should be a few seconds shorter than the grace period. SIGTERM → exit, with nothing lost SIGTERM signal handler runs stop intake fail readiness, refuse new work drain finish in-flight requests flush providers' shutdown() exit before the grace period ends the flush timeout must be shorter than the grace period or SIGKILL arrives mid-flush and the last batch is lost
The final flush has to happen after work stops and before the platform's patience runs out.

Configuration options

Setting Where Typical Note
Signal handled application SIGTERM, SIGINT the platform's default is immediate exit
Drain before flush application always flushing first loses the drain's telemetry
Providers shut down application all three each has its own queue
terminationGracePeriodSeconds pod spec 45 longest request plus flush
preStop sleep pod spec 5 s lets the load balancer remove the endpoint
Worker shutdown timeout prefork server below the grace period master waits for workers
atexit registration application also covers a normal exit with no signal

Verification

Prove the flush happens by looking for its evidence in a pod that has already gone.

# terminate a pod and read its final lines afterwards
kubectl delete pod checkout-7d9f8c5b6-xk2lm
kubectl logs checkout-7d9f8c5b6-xk2lm --previous --tail=5

# and confirm the spans from that window actually arrived
# (query the backend for spans whose service.instance.id matches the dead pod)

Expected Output: the flush line, and spans in the backend timestamped after the termination signal.

2026-09-18T14:12:38Z INFO  drain complete
2026-09-18T14:12:38Z INFO  telemetry flushed reason=lifespan
spans stored after SIGTERM: 47

Zero spans after the signal, with a flush line present, means the flush ran before the drain — the ordering failure from step 2, which is invisible in the log and obvious in this query.

Common mistakes

Flushing before draining. Error signature: a clean shutdown with no telemetry describing it. Root cause: providers shut down at the moment the signal arrives. Remediation: drain first; flush in the lifespan's exit path or after the server loop returns.

Only the tracer provider is flushed. Error signature: spans from the final seconds present, metrics and logs missing. Root cause: three providers, one shutdown call. Remediation: shut down all three, guarding each so one failure does not skip the others.

A grace period shorter than the drain. Error signature: the process killed mid-flush, with partial telemetry. Root cause: the platform's timeout expiring during the request drain. Remediation: set the grace period from the service's own latency tail plus a second for the flush.

Relying on atexit alone. Error signature: flushes that work locally and never in production. Root cause: atexit does not run when the process is terminated by a signal. Remediation: install a signal handler and register the same idempotent function with atexit.

No evidence the flush ran. Error signature: an argument about whether telemetry was lost or never produced. Root cause: the flush is silent. Remediation: log a line on completion, and treat its absence in a terminated pod's log as the diagnosis.

Frequently Asked Questions

Why does my service lose its last few seconds of telemetry on every deploy?

Because the process exits with records still in the batch processor's queue. The processor exports on an interval, so anything produced since the last export is in memory, and nothing flushes it unless the application calls shutdown explicitly.

Does atexit handle this?

Only for a clean interpreter exit. A process terminated by a signal does not run atexit handlers unless a signal handler causes a normal exit, and a process killed with SIGKILL runs nothing at all. Install a signal handler and treat atexit as a secondary path.

How long should the termination grace period be?

Long enough for the longest in-flight request to finish plus the flush. Thirty to forty-five seconds suits most services; the flush itself is usually under a second against a local collector, so the request drain dominates.

What about short-lived jobs and Lambda functions?

They need the flush structurally rather than defensively, because their whole lifetime can be shorter than one export interval. Call shutdown at the end of the job, or use a simple span processor that exports synchronously if the volume is small.

Related Guides