Tracing gRPC Services in Python

gRPC has no URLs, no query strings and no HTTP headers as far as your code is concerned — but it has metadata, and that is enough for trace context to travel. This page covers instrumenting grpcio servers and clients, the status mapping that decides whether your error rate means anything, and the streaming case where the default span shape stops working. It builds on instrumenting Python web frameworks, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "grpcio>=1.62.0,<2.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-instrumentation-grpc>=0.48b0,<1.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

export OTEL_SERVICE_NAME=orders-grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Implementation

Step 1 — Instrument the server. The instrumentor installs an interceptor; a process that both serves and calls needs the client instrumentor as well, and neither implies the other.

import grpc
from concurrent import futures
from opentelemetry.instrumentation.grpc import GrpcInstrumentorServer, GrpcInstrumentorClient
from observability.tracing import build_provider

build_provider()
GrpcInstrumentorServer().instrument()              # incoming calls
GrpcInstrumentorClient().instrument()              # outgoing calls from this process

server = grpc.server(futures.ThreadPoolExecutor(max_workers=16))
orders_pb2_grpc.add_OrdersServicer_to_server(OrdersServicer(), server)
server.add_insecure_port("[::]:50051")
server.start()
server.wait_for_termination()

Step 2 — Map the status deliberately. The interceptor records the gRPC status code; whether that counts as an error is a decision, and getting it wrong makes every error-rate panel meaningless.

import grpc
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

SERVER_FAULT = {
    grpc.StatusCode.INTERNAL,
    grpc.StatusCode.UNAVAILABLE,
    grpc.StatusCode.DEADLINE_EXCEEDED,
    grpc.StatusCode.RESOURCE_EXHAUSTED,
    grpc.StatusCode.DATA_LOSS,
}

def set_status(span, code: grpc.StatusCode) -> None:
    span.set_attribute("rpc.grpc.status_code", code.value[0])
    if code in SERVER_FAULT:
        span.set_status(Status(StatusCode.ERROR, code.name))
    else:
        span.set_status(Status(StatusCode.UNSET))   # NOT_FOUND is an answer, not a failure

NOT_FOUND means the server successfully determined the thing does not exist. Recording it as an error turns a normal cache-miss pattern into a permanent 30% error rate that everyone learns to ignore.

Step 3 — Add what the handler knows.

from opentelemetry import trace

class OrdersServicer(orders_pb2_grpc.OrdersServicer):
    def GetOrder(self, request, context):
        span = trace.get_current_span()             # the server span from the interceptor
        span.set_attribute("order.id", request.order_id)
        span.set_attribute("tenant.id", _tenant_from(context))

        order = repository.fetch(request.order_id)
        if order is None:
            context.abort(grpc.StatusCode.NOT_FOUND, "no such order")
        return order.to_proto()

Step 4 — Handle long streams with links. A span only reaches the backend when it ends.

from opentelemetry import trace
from opentelemetry.trace import Link

tracer = trace.get_tracer(__name__)

def Watch(self, request, context):
    stream_span = trace.get_current_span()
    stream_ctx = stream_span.get_span_context()

    for batch in subscription(request):
        with tracer.start_as_current_span(
            "orders.Watch/batch",
            links=[Link(stream_ctx)],               # linked to the stream, not nested in it
        ) as span:
            span.set_attribute("messages.count", len(batch))
            yield from batch

Step 5 — Exclude the infrastructure services. Health checking and reflection are high-volume and diagnostically empty.

export OTEL_PYTHON_GRPC_EXCLUDED_SERVICES="grpc.health.v1.Health,grpc.reflection.v1alpha.ServerReflection"

Configuration options

Verification

grpcurl -plaintext -d '{"order_id": 42}' localhost:50051 orders.Orders/GetOrder

Expected Output (Collector debug exporter):

Span #0
    Name           : /orders.Orders/GetOrder
    Kind           : Client
    Attributes:
         -> rpc.system: Str(grpc)
         -> rpc.service: Str(orders.Orders)
         -> rpc.method: Str(GetOrder)
Span #1
    Name           : /orders.Orders/GetOrder
    Kind           : Server
    Parent ID      : 00f067aa0ba902b7
    Attributes:
         -> rpc.grpc.status_code: Int(0)
         -> order.id: Int(42)

The property to check is Parent ID on the server span: it must equal the client span's ID, and both must share a trace ID. If the server span has no parent, metadata propagation is broken — check for a proxy stripping keys before checking anything in the code.

Then confirm the status mapping:

grpcurl -plaintext -d '{"order_id": 999999}' localhost:50051 orders.Orders/GetOrder

Expected Output:

    Status         : Unset
    Attributes:
         -> rpc.grpc.status_code: Int(5)

Status code 5 is NOT_FOUND, and the span status is Unset rather than Error — which is what keeps the service's error rate meaningful.

Common mistakes

The server span has no parent

Error signature: client and server traces are separate, each with one span. Root cause: either the client channel has no interceptor, or something between them strips the traceparent metadata key. Remediation: confirm GrpcInstrumentorClient().instrument() runs in the calling process, then check the mesh or proxy configuration. The propagation mechanics are in context propagation and baggage.

The error rate is permanently high

Error signature: 30% of spans have error status and nobody investigates any more. Root cause: NOT_FOUND and ALREADY_EXISTS are recorded as errors. Remediation: set error status only for server-fault codes.

A streaming span never appears

Error signature: a long-lived subscription produces no spans at all until the client disconnects. Root cause: one span covers the whole stream and a span only exports when it ends. Remediation: emit short per-batch spans linked to the stream span.

Attributes worth setting

The instrumentation records the RPC semantic conventions — system, service, method, status code — and stops there, which is correct because it knows nothing about your domain. Four additions carry most of the diagnostic value.

The deadline remaining. gRPC calls carry a deadline that shrinks across hops, and recording what was left when the call started makes a cascade of DEADLINE_EXCEEDED readable: the hop where the remaining budget first went negative is the cause, and the ones above it are consequences. Nothing else in the trace shows this.

The peer identity. For a server span, which client called; for a client span, which backend answered. In a service mesh with load balancing, the second is what distinguishes "the dependency is slow" from "one instance of the dependency is slow", and they have very different remedies.

The message size. Request and response sizes in bytes, which explain a slow call that is not slow for any interesting reason — it moved four megabytes. gRPC's own metrics carry this too, but on the span it is joined to the specific call.

Domain identity. The entity being operated on, the tenant, the operation's own outcome where it differs from the status code. Span attributes are the right place for high-cardinality values, so an entity ID here is fine and the same value as a metric label would not be.

def GetOrder(self, request, context):
    span = trace.get_current_span()
    remaining = context.time_remaining()
    if remaining is not None:
        span.set_attribute("rpc.grpc.deadline_remaining_ms", int(remaining * 1000))
    span.set_attribute("rpc.grpc.peer", context.peer())
    span.set_attribute("order.id", request.order_id)
    ...

Deadlines are the thing to get right

More than in HTTP, deadline handling is where gRPC services fail interestingly, and the tracing setup either makes that legible or does not.

Three practices help. Propagate the deadline rather than setting a fresh one per hop, so the budget genuinely shrinks and a downstream service cannot outlive the client waiting for it. Record what remained, as above. And treat DEADLINE_EXCEEDED on a client span differently from the same code on a server span: on the client it means the call did not finish in time, which may be the server's fault or may be that the budget was already exhausted on arrival; on the server it means the server itself gave up. Distinguishing them is what turns a cascade into a single identifiable cause.

One caveat about attribute volume: a gRPC service typically handles far more calls per second than an HTTP one, because the calls are cheaper and the clients are usually other services rather than browsers. Attributes that are affordable at web request rates can be a meaningful share of span size here, so the message-size and peer attributes are worth keeping and a full request payload is not.

Testing propagation across the boundary

The most valuable test for a gRPC service is not a unit test of an interceptor but an integration check that a real call produces a joined trace. It is cheap to write with an in-process server and an in-memory span exporter, and it catches every version of the failure that matters: a channel built without the interceptor, an interceptor registered in the wrong order, a metadata key that a middleware rewrote.

def test_client_and_server_spans_share_a_trace(in_memory_exporter, grpc_stub):
    grpc_stub.GetOrder(orders_pb2.GetOrderRequest(order_id=42))

    spans = in_memory_exporter.get_finished_spans()
    client = next(s for s in spans if s.kind is trace.SpanKind.CLIENT)
    server = next(s for s in spans if s.kind is trace.SpanKind.SERVER)
    assert client.context.trace_id == server.context.trace_id
    assert server.parent.span_id == client.context.span_id

Two assertions, and between them they cover the entire propagation path. Running it in CI means a dependency upgrade that changes interceptor behaviour fails the build rather than quietly disconnecting every trace in production.

Related

• Instrumenting Python web frameworks — the parent guide: server spans across frameworks.
• Instrumenting Flask with OpenTelemetry — the HTTP equivalent, including the prefork detail.
• Context propagation and baggage — how traceparent reaches the metadata in the first place.
• W3C Trace Context vs B3 propagation — what to do when the other side speaks a different format.
• Recording exceptions and span events — attaching a failure to the span that saw it.

Frequently Asked Questions