Context Variables and Thread Safety in Python Logging

Concurrent Python services need request-scoped state - a trace ID, a tenant, a user - to reach every log line without being threaded through every function signature. The naive answer, threading.local(), quietly corrupts that state the moment a single OS thread multiplexes many coroutines. This guide explains why contextvars is the correct primitive, how its token-based lifecycle works, how the logging module's own locks interact with it, and how to propagate context safely across coroutines, thread pools, and process boundaries. It is part of the Python Logging Fundamentals and Structured Data reference and pairs with the focused walkthroughs on using contextvars for request tracing and thread-safe logging in multiprocessing, with the emitted format itself covered in structured logging with the standard library.

Prerequisites

contextvars ships in the standard library from Python 3.7, so the core mechanics need no installation. The examples assume Python 3.11 or newer, which is where asyncio.TaskGroup and the context= argument to asyncio.create_task() become available. For the framework integration sections, pin a web framework and, optionally, the OpenTelemetry API used to seed real trace identifiers:

pip install \
  "starlette>=0.37.0,<1.0.0" \
  "opentelemetry-api>=1.30.0,<2.0.0"

Set the service identity so emitted records carry a consistent resource label, and make the fallback identifier explicit rather than implicit:

export OTEL_SERVICE_NAME="orders-api"
export LOG_MISSING_TRACE_ID="no-trace"

Nothing below requires an OpenTelemetry SDK: the context mechanics are pure standard library, and the tracing packages only matter when you want the identifiers in your logs to match the ones on your spans, as described in adding trace IDs to log records.

Concept & architecture

A ContextVar is a named slot whose value is resolved against the current context rather than the current thread. The current context is an immutable mapping that the interpreter snapshots and restores automatically at coroutine switch points. That is the whole reason contextvars exists: under asyncio, one OS thread interleaves thousands of requests, so any state keyed by thread identity is shared across all of them. A value written by request A would be read by request B the instant the loop switched, producing trace ID collisions and cross-request data leaks.

Binding state to the execution context instead gives each logical flow its own isolated view. When asyncio.create_task() schedules a coroutine, it copies the current context, so the child starts with the parent's values but its own mutations stay private. This is structural isolation - no locks, no manual save-and-restore.

It is worth being precise about what "the current context" is. At any instant the interpreter holds an immutable Context mapping ContextVar objects to values, implemented as a hash array mapped trie so that layering a new value shares structure with the old one instead of copying it. A get() is a lookup in that mapping; a set() does not mutate the mapping in place but installs a new value and hands back a Token describing how to undo it. Because the mapping is immutable, snapshotting it is cheap and safe to share - and that immutability is precisely what makes automatic copying across coroutine boundaries correct without any locking. Two tasks holding snapshots of the same context cannot corrupt each other's view of a scalar value, because neither can rewrite the other's mapping.

The lifecycle is token-based and deliberately explicit. ContextVar.set(value) returns a Token whose old_value records the previous state, or the Token.MISSING sentinel if the variable was unset. Passing that token to ContextVar.reset(token) restores the prior value. The discipline matters: because each set() layers a new mapping, a request that sets but never resets leaves that layer in place, and in a long-lived worker those layers accumulate into unbounded growth. A token is also single-use and context-bound - reusing one raises RuntimeError: ... has already been used once, and resetting in a different context raises ValueError: ... was created in a different Context. Always pair a set() with a reset() in a finally block, or wrap the pair in a context manager.

One boundary breaks the automatic copying: code that leaves the event loop. A concurrent.futures.ThreadPoolExecutor runs work on a thread the loop did not create through create_task, so it does not inherit the caller's context. The fix is to snapshot the live context with contextvars.copy_context() and execute the work through Context.run(), which restores that snapshot inside the worker - which is exactly what asyncio.to_thread() does for you, and exactly what loop.run_in_executor() does not.

The process boundary deserves emphasis because it is the most common misconception. A ContextVar lives inside one interpreter. When a ProcessPoolExecutor or a multiprocessing worker starts, the child either forks - inheriting a copy of memory frozen at fork time, not the value set later in the request - or spawns fresh with no inherited values at all. Either way, the running request's identifiers do not travel to the child automatically. To carry a trace ID across a process you must serialize it into the work item, the queue payload, or the network request, and re-establish the ContextVar inside the child before it logs. Treating cross-process propagation as a serialization problem rather than a context problem is the mental model that keeps it correct.

Architecturally, the point of all this is a clean three-layer split inside the logging pipeline. The ContextVar owns where the value lives, a logging.Filter owns when it is attached to a record, and a logging.Formatter owns how it is serialized. Keeping those separate means the same context layer feeds a JSON formatter in production and a console formatter locally without changing a line, which is the layering used throughout formatter configuration.

Step-by-step implementation

1. Declare the context variables at module scope. Define each ContextVar once, with a sentinel default, so every importer shares the same slot. Creating them inside a function makes a new, unrelated variable on every call - the values you set will never be visible to the filter that reads the module-level one. An explicit default also means get() returns a fallback instead of raising LookupError on any code path that runs outside a request.

import contextvars

request_id_ctx: contextvars.ContextVar[str | None] = contextvars.ContextVar(
    "request_id", default=None
)
trace_id_ctx: contextvars.ContextVar[str] = contextvars.ContextVar(
    "trace_id", default="0" * 32
)

2. Set the values at the request boundary and reset on teardown. Middleware is the natural place: it owns the full request lifecycle, so it can guarantee the matching reset. Capture the token in the same scope as the set, and never store a token on self or in a module global - it is valid only in the context that produced it.

import uuid
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response


class ContextMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next) -> Response:
        token = request_id_ctx.set(
            request.headers.get("X-Request-ID") or str(uuid.uuid4())
        )
        try:
            return await call_next(request)
        finally:
            # reset runs whether the handler succeeds or raises
            request_id_ctx.reset(token)

3. Wrap the set/reset pair once so no call site can forget it. A three-line context manager removes the entire class of missing-reset bugs and makes the scope of an identifier obvious at the call site. Use it anywhere outside middleware - background jobs, consumers, tests.

import contextlib
from collections.abc import Iterator


@contextlib.contextmanager
def scoped(var: contextvars.ContextVar, value) -> Iterator[None]:
    token = var.set(value)  # token stays local to this frame
    try:
        yield
    finally:
        var.reset(token)  # runs on success, exception, or generator close

4. Read the context inside a logging filter, never on the handler. A handler instance is long-lived and shared, so caching a value on it would freeze the first request's identifiers onto every later record. Resolve inside filter() so each record reflects the context active at emission, and return True unconditionally - this filter enriches, it does not drop.

import logging


class ContextFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_ctx.get() or "no-request"
        record.trace_id = trace_id_ctx.get()
        return True

5. Propagate explicitly across the thread-pool boundary. Snapshot the caller's context and run the offloaded callable through it so the worker thread sees the same identifiers. Snapshot once per submission batch, not once per item, since copy_context() allocates.

import concurrent.futures


def run_in_context(executor, fn, *args):
    ctx = contextvars.copy_context()  # snapshot the caller's context
    return executor.submit(ctx.run, fn, *args)

6. Attach the filter to the handler, not the logger, when you want uniform output. Attaching the ContextFilter to a handler means every record that handler emits is enriched, regardless of which logger produced it - including records from third-party libraries that propagate up to the root. Attaching it to a logger enriches only that logger's own records and, crucially, does not enrich records propagated from its children.

import logging

handler = logging.StreamHandler()
handler.addFilter(ContextFilter())  # enriches every record this handler emits
logging.getLogger().addHandler(handler)

In a declarative setup the same wiring goes through dictConfig, where filters are named objects attached to a handler's filters list; see logging configuration and dictConfig for the full schema. The separation - context lives in a ContextVar, enrichment lives in a Filter, serialization lives in a Formatter - is what keeps each piece independently testable and reusable across handler architectures.

Configuration reference

The knobs here are API surfaces rather than environment variables, so the useful reference is what each call returns, what it does by default, and what a production service should do with it.

The second thing worth keeping in one place is which boundaries copy context and which do not, because every propagation bug traces back to a row in this table.

Async & concurrency considerations

The split that trips teams up is between coroutines and threads. asyncio.create_task(), asyncio.gather(), and asyncio.TaskGroup all copy the current context for the new task, so a trace ID set before the spawn is visible inside it automatically - and changes made inside the task do not leak back to the parent. That is exactly the isolation you want for per-request state, and it is why the async tracing patterns used for spans and for log context are the same mechanism underneath.

A thread pool is different, and the two ways of reaching one behave differently. asyncio.to_thread() copies the current context internally and runs your callable through it, so identifiers arrive intact. loop.run_in_executor() and executor.submit() hand work to a thread the loop did not create as a task, so no copy happens and the worker runs under whatever context that thread last held - usually the default. The difference is visible in three lines:

import asyncio
import concurrent.futures
import contextvars

trace_id_ctx = contextvars.ContextVar("trace_id", default="DEFAULT")


def peek(label: str) -> None:
    print(label, trace_id_ctx.get())


async def main() -> None:
    trace_id_ctx.set("trace-1")
    await asyncio.to_thread(peek, "to_thread:")          # copies context
    loop = asyncio.get_running_loop()
    with concurrent.futures.ThreadPoolExecutor() as pool:
        await loop.run_in_executor(pool, peek, "run_in_executor:")  # does not
        ctx = contextvars.copy_context()                  # so snapshot manually
        await loop.run_in_executor(pool, ctx.run, peek, "ctx.run:")


asyncio.run(main())

Expected Output:

to_thread: trace-1
run_in_executor: DEFAULT
ctx.run: trace-1

Generators are the second surprise, and the folklore about them is wrong in a way worth stating plainly: a generator does not capture a context when it is created. Its body executes in whatever context is current at each next(), and a set() inside the generator body leaks out to the caller, because PEP 567 deliberately shipped without per-generator contexts. So a long-lived generator or an async iterator that sets a variable mid-stream will silently mutate the context of whoever is draining it. The safe pattern is to keep set() calls out of generator bodies entirely, or to wrap the mutation in the scoped() context manager from step three so it unwinds at the yield.

Background schedulers warrant a deliberate convention. A periodic job, a consumer pulling from a broker, or a retry worker has no inbound request to inherit from, so it should mint a fresh trace ID at the start of each unit of work and reset it when that unit completes. Anchoring the set/reset pair to the boundary of one job - one message, one tick, one batch - keeps the context mapping flat and gives every emitted log line an identifier you can group on, even though no HTTP request was ever involved. For genuinely detached work, pass context=contextvars.Context() to create_task() so the job starts from a clean slate rather than silently inheriting whichever request happened to schedule it.

Thread safety of the logging module itself is a separate guarantee that people conflate with context isolation. logging serializes emission with a module-level lock plus a per-handler RLock, so two threads writing through the same handler cannot interleave half-formatted lines. That safety has costs and edges. The cost is contention: every record through a shared handler acquires that handler's lock, so a slow sink - a network socket, a synchronous file flush - turns into a queue in front of your request threads. The standard remedy is to move the write off the caller entirely with non-blocking logging via QueueHandler, keeping only an enqueue on the hot path. The edge is fork: threads do not survive it, so a forked child inherits handler objects whose QueueListener thread no longer exists, and records enqueued in the child are never drained. CPython registers fork hooks that re-initialize logging's locks in the child, which prevents the classic deadlock, but it cannot resurrect your listener thread - restart it in a post-fork hook, as covered in thread-safe logging in multiprocessing.

Free-threaded builds change the performance picture without changing the semantics. contextvars remains correct because it was never keyed on thread identity, and the copy rules are identical. What changes is that handler locks now serialize genuinely parallel threads instead of threads that were already taking turns under the GIL, so a heavyweight formatter or a chatty handler becomes a visible bottleneck rather than a hidden one. The same advice applies, only harder: keep enrichment cheap, keep the handler's emit short, and push serialization and I/O behind a queue.

There is also a measurable cost worth budgeting for. On CPython 3.12+, a get() is a few tens of nanoseconds and a set() is roughly twice that, so the rule of thumb is one set() per logical scope rather than per log line. In a request handler that means setting identifiers once in middleware and letting hundreds of downstream log calls read them. Snapshotting via copy_context() adds a few microseconds per thread-pool submission - immaterial next to the thread handoff itself, but a reason not to snapshot inside a tight loop. These costs are small enough that correctness, not performance, should drive the design; the failure mode that actually hurts is unbounded context growth from missing resets, not lookup latency.

Production code examples

End-to-end: filter injection with safe scoping

This module wires a ContextVar into a JSON-emitting logger and demonstrates the set-and-reset discipline a request handler must follow.

# Tested with Python 3.11+ (standard library only)
import contextvars
import json
import logging

request_id_ctx: contextvars.ContextVar[str | None] = contextvars.ContextVar(
    "request_id", default=None
)


class ContextFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        # Resolved per record, so each line reflects the live context.
        record.request_id = request_id_ctx.get() or "no-request"
        return True


class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        return json.dumps({
            "ts": self.formatTime(record),
            "level": record.levelname,
            "msg": record.getMessage(),
            "request_id": getattr(record, "request_id", None),
        })


logger = logging.getLogger("app")
logger.setLevel(logging.INFO)
_handler = logging.StreamHandler()
_handler.setFormatter(JSONFormatter())
_handler.addFilter(ContextFilter())  # on the handler: enriches everything it emits
logger.addHandler(_handler)


def handle_request(req_id: str) -> None:
    token = request_id_ctx.set(req_id)
    try:
        logger.info("processing request")
    finally:
        request_id_ctx.reset(token)  # never leaks into the next request


if __name__ == "__main__":
    handle_request("req-8842")
    logger.info("idle heartbeat")  # outside any request scope

Expected Output:

{"ts": "2026-07-25 12:00:00,000", "level": "INFO", "msg": "processing request", "request_id": "req-8842"}
{"ts": "2026-07-25 12:00:00,001", "level": "INFO", "msg": "idle heartbeat", "request_id": "no-request"}

The second line is the point of the sentinel default: work that runs outside a request is labelled honestly instead of inheriting a stale identifier.

End-to-end: an async service with a thread-pool offload

The pattern that exercises every boundary at once is an async service that offloads CPU-bound work to a thread pool while keeping logs correlated. The middleware-equivalent sets the trace ID, async handlers inherit it for free, and the offloaded work re-establishes it explicitly.

# Tested with Python 3.11+ (standard library only)
import asyncio
import concurrent.futures
import contextvars
import json
import logging

trace_id_ctx: contextvars.ContextVar[str] = contextvars.ContextVar(
    "trace_id", default="no-trace"
)
_pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)


class TraceFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.trace_id = trace_id_ctx.get()
        return True


class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        return json.dumps({
            "level": record.levelname,
            "msg": record.getMessage(),
            "trace_id": getattr(record, "trace_id", "no-trace"),
        })


log = logging.getLogger("svc")
log.setLevel(logging.INFO)
_h = logging.StreamHandler()
_h.setFormatter(JSONFormatter())
_h.addFilter(TraceFilter())
log.addHandler(_h)


def cpu_work(n: int) -> int:
    # Runs on a worker thread; reads the trace ID restored by ctx.run.
    log.info(f"hashing batch {n}")
    return n * n


async def handle(trace_id: str) -> int:
    token = trace_id_ctx.set(trace_id)
    try:
        log.info("request received")  # inherits trace_id directly
        loop = asyncio.get_running_loop()
        ctx = contextvars.copy_context()  # snapshot before crossing to a thread
        result = await loop.run_in_executor(_pool, lambda: ctx.run(cpu_work, 7))
        log.info("request done")
        return result
    finally:
        trace_id_ctx.reset(token)


async def main() -> None:
    await asyncio.gather(handle("trace-aaa"), handle("trace-bbb"))


if __name__ == "__main__":
    asyncio.run(main())
    _pool.shutdown()

Expected Output:

{"level": "INFO", "msg": "request received", "trace_id": "trace-aaa"}
{"level": "INFO", "msg": "hashing batch 7", "trace_id": "trace-aaa"}
{"level": "INFO", "msg": "request done", "trace_id": "trace-aaa"}
{"level": "INFO", "msg": "request received", "trace_id": "trace-bbb"}
{"level": "INFO", "msg": "hashing batch 7", "trace_id": "trace-bbb"}
{"level": "INFO", "msg": "request done", "trace_id": "trace-bbb"}

The line ordering is scheduling-dependent and will interleave differently between runs; the field values are not. The two concurrent requests never cross-contaminate: each handle coroutine carries its own isolated trace_id, the async log lines inherit it automatically, and the thread-pool log lines carry it only because ctx.run restored the snapshot. Drop the copy_context() and the hashing batch lines fall back to no-trace, which is the exact symptom that flags a missing boundary snapshot in production. Replacing the run_in_executor call with await asyncio.to_thread(cpu_work, 7) produces identical output with no snapshot at all - worth preferring whenever the default executor is acceptable.

Common mistakes

• Error signature: RuntimeError: <Token ...> has already been used once, or ValueError: <Token ...> was created in a different Context. Root cause: the token outlived its frame - stored on self, cached in a module global, or reset from inside a ctx.run() block or a different task than the one that set the value. Remediation: treat a token as a local variable with the same lifetime as the try block that follows it, and hide the pair behind a @contextlib.contextmanager helper so no call site can reuse or relocate it.

• Error signature: log lines from offloaded work carry no-trace or no-request while the surrounding async lines carry the correct ID. Root cause: loop.run_in_executor() or executor.submit() was used directly, and neither copies the caller's context. Remediation: switch to asyncio.to_thread(), or snapshot with contextvars.copy_context() and submit ctx.run as the callable, as in the worker-pool example above.

• Error signature: every log line in the service shares one trace ID - usually whichever request happened to arrive first after a deploy. Root cause: the context value was read once and cached on a long-lived object, typically in a handler's or adapter's __init__. Remediation: resolve the ContextVar inside Filter.filter(), which runs once per record, and never in a constructor or at module import.

• Error signature: resident memory climbs slowly in a long-lived worker and never returns to baseline between requests. Root cause: set() without a matching reset(), so each request leaves another context layer behind. Remediation: always reset in finally, and add a test that asserts the variable equals its default after a simulated request completes - that single assertion catches the whole family of leaks.

• Error signature: two concurrent requests see each other's fields even though each set its own value. Root cause: the ContextVar holds a mutable object - a dict of log fields, a list of tags - and both flows are mutating the same referenced object; the slot is isolated, the value is not. Remediation: store immutable values, or replace the whole object on every update (ctx.set({**ctx.get(), "k": v})) so mutation never happens in place. The same discipline applies to bound loggers, which is why binding context variables in structlog rebinds rather than mutates.

Related reading

• Python Logging Fundamentals and Structured Data - the parent reference covering handlers, formatters, levels, and configuration.
• Using contextvars for Request Tracing in Python - the end-to-end middleware-to-log-line walkthrough.
• Thread-Safe and Multiprocessing-Safe Logging in Python - what happens to handlers, locks, and context after a fork.
• Python Formatter Configuration for Production Observability - turning the enriched record into a stable JSON contract.
• Handler Architecture for Python Logging - where the locks live and how to keep the write off the request path.
• Context Propagation and Baggage in Python OpenTelemetry - the same context mechanics applied to cross-service trace headers.
• Logging from asyncio tasks without blocking — keeping a slow sink off the event loop while the records keep their request context.

Frequently Asked Questions