Telemetry from Serverless and Batch Python

The usual telemetry machinery assumes a process that lives for hours: it batches on a schedule, it is scraped on an interval, and it has a stable identity to group by. A Lambda invocation, a cron job and a data pipeline step break all three assumptions, and the symptom is the same in every case — telemetry that works perfectly in the service and silently produces nothing here. This guide covers what has to change. It is part of the Python telemetry pipelines and delivery section. The focused articles in this topic are Logging from Cron and Batch Jobs, Metrics from Short-Lived Jobs with the Pushgateway and Tracing AWS Lambda Python Functions.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-http>=1.27.0,<2.0.0" \
            "prometheus-client>=0.20.0,<1.0.0"

Concept and architecture

Three assumptions break, and each has a specific remedy.

The export schedule assumes the process will still exist. BatchSpanProcessor collects spans and exports every few seconds from a background thread. A process that exits in three seconds never reaches its first scheduled export, and the thread is a daemon, so it does not delay the exit. The remedy is either an explicit flush at the end of the work, or a simple processor that exports on span end — which is inefficient at high volume and perfectly reasonable at the volumes a short job produces.

The scrape model assumes the process can be reached. Prometheus-style collection asks a process for its current values on an interval. A process that has already exited cannot be asked. The remedy is to push: to a gateway that holds the last values until the next scrape, or — often better — to record the outcome as span attributes or a structured log record and let the metrics be derived from those.

The identity model assumes a stable instance. A long-lived service has a host, a pod and an instance identifier that make sense as grouping dimensions. A job that runs every minute creates a new one each time, and using it as a label produces exactly the cardinality explosion described in controlling label cardinality in Prometheus. The remedy is to identify the job rather than the process: a stable job name as a label, and the run identifier as a span attribute or log field where cardinality is free.

There is also a fourth property that is not a broken assumption but a new requirement: a short-lived process that fails to start produces no telemetry whatsoever, so its absence is the only signal. Monitoring for the absence of a run is therefore not optional in a way it never is for a service.

Step-by-step implementation

Step 1 — Wrap the whole run in a context manager that flushes. Making the flush structural rather than remembered is the single most effective change, because the failure mode of forgetting is total and silent.

# jobtelemetry.py
import contextlib
import os
import uuid

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter


@contextlib.contextmanager
def job_telemetry(job_name: str):
    run_id = os.environ.get("RUN_ID") or uuid.uuid4().hex
    provider = TracerProvider(resource=Resource.create({
        "service.name": job_name,            # the JOB is the service, not the process
        "deployment.environment": os.environ.get("ENVIRONMENT", "dev"),
    }))
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(timeout=10)))
    trace.set_tracer_provider(provider)

    tracer = trace.get_tracer(job_name)
    with tracer.start_as_current_span(f"job {job_name}") as span:
        span.set_attribute("job.run_id", run_id)     # attribute, never a label
        span.set_attribute("job.attempt", int(os.environ.get("ATTEMPT", "1")))
        try:
            yield span
        finally:
            # 1. The delivery mechanism, not a safety net.
            provider.shutdown()

Step 2 — Use a simple processor where the volume is tiny. A job producing a few dozen spans gains nothing from batching and loses the risk of an unflushed queue. Exporting on span end makes delivery immediate and the failure mode visible, because an export failure raises where the work is rather than silently in a background thread.

Step 3 — Push metrics, or derive them from spans. A job's numbers — rows processed, duration, outcome — can reach a metrics store two ways. Pushing to an intermediary that holds values for the next scrape is the direct route. Recording them as span attributes and deriving metrics downstream avoids the intermediary entirely and keeps the numbers attached to the run that produced them, which is usually more useful.

with job_telemetry("nightly-reconciliation") as span:
    processed = run_reconciliation()
    # 2. Counts as attributes: no series, no cardinality, still queryable.
    span.set_attribute("job.rows_processed", processed.rows)
    span.set_attribute("job.rows_rejected", processed.rejected)
    span.set_attribute("job.outcome", "ok" if not processed.rejected else "partial")

Step 4 — Carry trace context in through the trigger. A job scheduled by a request should join that request's trace. The context travels in whatever the trigger carries: a message header, an event field, an environment variable set by the scheduler. Extracting it is the same operation as on any other boundary, covered for the queue case in propagating trace context across Celery tasks.

from opentelemetry.propagate import extract

# The scheduler put the W3C header into the job payload when it enqueued it.
carrier = {"traceparent": payload.get("traceparent", "")}
parent_context = extract(carrier)

with tracer.start_as_current_span("job step", context=parent_context):
    ...

Step 5 — Emit a heartbeat that says the run happened. The absence of a run is invisible from the run's own telemetry, so something has to record that it started and finished. A span is enough if the monitoring can alert on its absence; otherwise a single pushed metric with the completion timestamp is the conventional approach.

Step 6 — Keep per-item work out of the span count. A job processing a hundred thousand records should not produce a hundred thousand spans. It should produce one span for the run, a handful for its phases, and counts as attributes. The temptation to span each item is strong and produces a trace nobody can open.

Configuration reference

Async and concurrency considerations

Short-lived processes concentrate a problem that long-lived ones spread out: all of their telemetry is produced and delivered in a very short window, which makes the export part of the critical path in a way it normally is not.

For a synchronous job this is usually acceptable — an extra two hundred milliseconds on a forty second run is invisible. For a Lambda function billed by the millisecond it is a direct cost, and it is the reason the platform-provided extensions exist: they run alongside the function, accept the telemetry locally, and forward it after the response has been returned, so the invocation is not billed for the export. Where such an extension is available it is the right answer, and the application still needs to flush into it.

For an asyncio job the flush must happen after the loop's work completes but before the loop closes, which means inside the coroutine rather than after asyncio.run returns. Calling a synchronous provider shutdown from inside a coroutine blocks the loop for the duration of the export — acceptable at the end of a job, and a mistake anywhere else.

Concurrency within the job also affects span structure. A job that fans out across a thread pool must propagate context into the workers or their spans become roots, and the run's trace fragments into hundreds of unrelated traces. The mechanism is the same as for any pool boundary, covered in propagating context across thread and process pools.

Production code examples

A complete cron job with telemetry that is structurally impossible to forget:

#!/usr/bin/env python3
# nightly.py — a scheduled job, instrumented so the flush cannot be skipped.
import logging
import sys

from jobtelemetry import job_telemetry

log = logging.getLogger("nightly")


def main() -> int:
    with job_telemetry("nightly-reconciliation") as span:
        try:
            with trace.get_tracer(__name__).start_as_current_span("read"):
                rows = load_rows()
            with trace.get_tracer(__name__).start_as_current_span("transform"):
                cleaned, rejected = transform(rows)
            with trace.get_tracer(__name__).start_as_current_span("write"):
                written = write(cleaned)
        except Exception as exc:
            # 1. Record on the RUN span, so the failure is on the thing being monitored.
            span.record_exception(exc)
            span.set_attribute("job.outcome", "failed")
            log.exception("job failed")
            return 1

        span.set_attribute("job.rows_read", len(rows))
        span.set_attribute("job.rows_written", written)
        span.set_attribute("job.rows_rejected", len(rejected))
        span.set_attribute("job.outcome", "partial" if rejected else "ok")
        return 0


if __name__ == "__main__":
    sys.exit(main())

Expected Output: one compact trace per run, carrying everything a query needs.

{
  "name": "job nightly-reconciliation",
  "durationMs": 41208,
  "attributes": {
    "job.run_id": "0d41f2a8c9b74e1f",
    "job.attempt": 1,
    "job.rows_read": 100000,
    "job.rows_written": 99842,
    "job.rows_rejected": 158,
    "job.outcome": "partial"
  },
  "children": ["read", "transform", "write"]
}

An absence alert, which is the check no amount of in-job instrumentation can replace:

# the job has not completed within 90 minutes of its expected hourly schedule
time() - max by (job) (job_last_success_timestamp_seconds{job="nightly-reconciliation"}) > 5400

Expected Output: silence while the job runs, and a page when a run does not happen at all — which is the failure the job itself cannot report.

JobDidNotRun  firing  job=nightly-reconciliation  last success 6h12m ago

The shapes this covers, and how each one differs

"Short-lived" covers several quite different situations, and the remedies above apply to each with a different emphasis.

A function-as-a-service invocation is the extreme case. Its lifetime is measured in milliseconds to seconds, it is billed by duration so every millisecond of export is a direct cost, and — the property that catches people — the execution environment is frozen the instant the handler returns. A background thread scheduled to export in five seconds does not run five seconds later; it runs whenever the environment is next thawed for another invocation, which may be minutes away or never. This is why an export that appears to work in testing, where invocations are frequent, produces nothing in production where they are sparse. The platform's telemetry extension, where one exists, is the correct answer; an explicit flush before returning is the fallback.

A scheduled job runs for seconds to minutes, on a schedule, usually as a container or a host process. Its distinguishing property is that nobody is watching: nothing downstream notices if it does not run, and its failures are discovered when somebody asks why a report is stale. The absence alert matters more here than anywhere else, and it must be based on completion rather than on start, because a job that starts and hangs produces the same evidence as one that succeeded.

A queue worker task is short-lived logically but runs inside a long-lived process, which changes everything. The provider is initialised once and shared, the export schedule works normally, and the flush concern disappears. What remains is context propagation from the message and the discipline about span granularity, plus the per-worker considerations in collecting metrics from Celery workers. Treating a queue task as though it were a standalone short-lived process — initialising a provider per task — is a common and expensive mistake.

A data pipeline step runs for minutes to hours and is short-lived only relative to a service. Its telemetry mostly works by default, and its distinctive problem is volume: a step processing millions of rows will produce an unusable trace unless the granularity discipline in step 6 is applied deliberately.

Making the job's telemetry answer the right questions

A service's telemetry answers "what happened to this request". A job's telemetry should answer a different and smaller set of questions, and designing for them directly produces better results than adapting the service pattern.

Did it run? The most important question and the one the job cannot answer about itself. It needs a completion timestamp recorded somewhere durable, and an alert on that timestamp's age.

Did it finish, and how did it end? An outcome attribute with a small set of values — succeeded, partial, failed — is worth more than any amount of detail, because it is what a dashboard groups by and what an alert fires on. Deriving it from the presence or absence of an exception is less reliable than setting it explicitly, since a job can complete without raising and still have done the wrong thing.

How much did it do? Counts, as attributes: rows read, rows written, rows rejected, bytes transferred. These are the numbers that make a run comparable with yesterday's run, which is how a slow degradation is noticed. A job that processed ninety-nine thousand rows every night and forty thousand last night has a problem that no error message would have reported.

How long did each phase take? Child spans for the phases give this directly, and phase durations are far more actionable than a total, because a job that is slower overall is almost always slower in exactly one phase. Three or four phase spans per run is the right granularity: enough to localise a regression, few enough to read at a glance.

Which run was this? A run identifier on the span, matching whatever the scheduler calls it, so that a person looking at a scheduler's failed-run entry can find the corresponding trace without guessing from timestamps. This is a small thing that removes a genuinely annoying step from every investigation.

Common mistakes

No flush, and therefore no telemetry. Error signature: a job that runs successfully and appears nowhere. Root cause: the process exits before the first scheduled export. Remediation: flush structurally, in a context manager around the whole run.

Metrics that are never scraped. Error signature: a metrics endpoint that exists and is never read. Root cause: scrape-based collection against a process that has already exited. Remediation: push, or carry the numbers as span attributes.

The run identifier used as a metric label. Error signature: series count growing by one per run, forever. Root cause: a unique value in a label. Remediation: job name as the label, run identifier as an attribute.

One span per processed item. Error signature: traces that no viewer will open and an export measured in hundreds of megabytes. Root cause: instrumenting the loop rather than the run. Remediation: counts as attributes, spans for phases, per-item spans only for failures.

No alert on the job not running. Error signature: a scheduled task that has been failing to start for three weeks. Root cause: monitoring that depends on the job producing telemetry. Remediation: an absence check based on the last successful completion timestamp.

A provider created per task inside a worker. Error signature: memory climbing steadily in a long-running worker, and one export connection per task. Root cause: treating a logically short-lived unit of work as a short-lived process. Remediation: initialise the provider once when the worker starts, and create only spans per task.

Logging configured per run with no deduplication. Error signature: duplicated log records, multiplying with every invocation in a reused execution environment. Root cause: adding a handler to the root logger on every invocation, in an environment that persists between them. Remediation: configure logging once at module import, guarded so a second call is a no-op.

The scheduler's retry and the job's retry both instrumented as one run. Error signature: a run whose duration is the sum of three attempts and whose outcome is the last one. Root cause: no attempt number, so three separate executions are indistinguishable from one long one. Remediation: record the attempt explicitly, as the context manager above does, so a query can count attempts and successes separately.

Context dropped at the trigger. Error signature: background work appearing as unrelated root traces. Root cause: no propagation through the message or event. Remediation: inject at enqueue, extract at start, as in step 4.

Frequently Asked Questions