Logging for Python Library Authors

An application's logging is its own. A library's logging is a guest in someone else's process, and the conventions for library logging exist so that every application importing the library keeps complete control over what appears, where it goes and how it looks. Getting it wrong is easy and quietly damaging: a library that configures the root logger changes every application's output format; one that logs routine events at warning floods every service that uses it; one that logs request bodies leaks data it had no way to know was sensitive. This page covers the conventions. It is a task article under logging in Python runtimes and frameworks, part of the modern Python logging libraries deep dive section, and it is the other side of taming third-party library loggers.

Prerequisites

The standard library is all a library should depend on for logging. Adding a third-party logging package as a library dependency imposes it on every application.

pip install "pytest>=8.0.0,<9.0.0"   # for the tests below

Implementation

Step 1 — Create loggers named after modules. logging.getLogger(__name__) at the top of each module produces loggers such as acme_client.http and acme_client.retry, all under the package's namespace. An application can then control the whole library with one entry for acme_client, or a single subsystem with acme_client.retry, without the library doing anything to enable it.

# acme_client/http.py
import logging

log = logging.getLogger(__name__)          # "acme_client.http"

Step 2 — Attach one NullHandler to the top logger. When an application has configured no logging at all, Python falls back to a last-resort handler that prints warnings and above to standard error. A NullHandler on the library's top-level logger means the library's records are discarded quietly in that case, and it has no effect whatsoever once the application configures logging.

# acme_client/__init__.py
import logging

logging.getLogger(__name__).addHandler(logging.NullHandler())

Step 3 — Never configure anything else. No handlers beyond that one, no formatters, no levels, and above all nothing on the root logger. A library cannot know whether the application logs JSON to standard output, text to a file, or records through OpenTelemetry. Any configuration it installs is wrong for most of its users.

# never, anywhere in a library
logging.basicConfig(level=logging.INFO)                 # takes over the root logger
logging.getLogger().addHandler(logging.StreamHandler()) # duplicates every record
logging.getLogger(__name__).setLevel(logging.DEBUG)     # overrides the application's choice

Step 4 — Choose levels for the application's operator. The people reading a library's warnings are operators of applications that use it, who did not write it and may not know its internals. Routine operation — a connection opened, a retry scheduled, a cache miss — is debug. An event an operator might want to see in production — a fallback endpoint in use, a deprecated option set — is info or warning. A warning should describe something the operator can act on; a library whose normal operation produces warnings trains every operator to ignore its warnings.

log.debug("retrying request", extra={"attempt": attempt, "delay_s": delay})
log.warning("primary endpoint unavailable, using fallback",
            extra={"primary": self._primary_host})

Step 5 — Keep secrets and payloads out. A library has no way to know which of its inputs its users consider sensitive, so it should assume all of them might be. Credentials, tokens and full request or response bodies do not belong in records. Identifiers, sizes, status codes and timings usually do, and give operators what they need without the risk.

# useful and safe
log.debug("request complete", extra={"method": method, "path": path,
                                     "status": resp.status, "bytes": len(resp.content),
                                     "duration_ms": elapsed_ms})

# never — the library cannot know what the body contains
log.debug("request body: %s", body)

Step 6 — Do not log exceptions you raise. An exception propagated to the caller will be handled or logged by the caller. Logging it inside the library as well produces two records for one failure, one of them without the application's context. Log only exceptions the library catches and handles internally — a retried timeout, a fallback taken — and let everything else carry its information in the exception itself.

Making a library's logs useful to its users

Correct conventions stop a library from causing harm. A few further habits make its logs genuinely helpful to the people debugging applications that use it.

Use structured fields rather than formatted strings. Passing details through extra means applications with JSON formatters get queryable fields, and applications with text formatters lose nothing. A message like "request to %s failed after %d attempts" is readable; the same information as fields — host, attempts — is also filterable, which matters when an operator is trying to find every retry against one endpoint.

Keep messages constant. A fixed message with variable data in fields groups correctly in any log store, so an operator can count how often something happens. A message with values interpolated into it produces a distinct string per occurrence, and grouping fails.

Log decisions, not just events. "Using fallback endpoint because primary returned 503 three times" is far more useful to an operator than three records saying a request failed and one saying a request succeeded. The library knows why it did what it did; the application does not, unless the library says so.

Name the subsystems sensibly. Module-based logger names are the default and usually right. Where a module contains several independently interesting behaviours — connection management and retries, for instance — separate child loggers let operators enable debug output for exactly the behaviour they are investigating, which is the capability described in changing log levels at runtime.

Document the logger names. A short section in the library's documentation listing its loggers and what each one covers turns "why is this library slow" into a configuration change rather than a source-code reading exercise.

Libraries and the other signals

The same principle — emit, do not configure — applies to tracing and metrics, and it matters just as much there.

A library that wants to produce spans should depend on the OpenTelemetry API, not the SDK, and obtain a tracer through the global provider. If the application has configured the SDK, the library's spans join the application's traces automatically; if it has not, the API returns a no-op tracer and the library's instrumentation costs almost nothing. A library that creates its own tracer provider, or exports spans itself, has made the same mistake as one that calls basicConfig: it has decided on behalf of every application where telemetry goes.

Metrics follow the same shape. A library that records metrics should use the OpenTelemetry metrics API or accept a registry from the application, rather than registering collectors in a global registry or starting its own metrics endpoint. The first leaves the decision with the application; the second creates series and endpoints the application did not ask for and cannot easily remove. The broader context on instruments is in recording counters and histograms with OpenTelemetry.

The pattern across all three signals is that a library should be instrumented and never configured. Instrumentation — log calls, spans, metric recordings — describes what the library does. Configuration — handlers, exporters, providers, registries — decides what happens to that description, and that decision belongs to the application alone.

Configuration options

Verification

Test that importing the library changes nothing about the application's logging.

import logging

def test_import_does_not_configure_logging():
    root = logging.getLogger()
    before = list(root.handlers), root.level

    import acme_client  # noqa: F401

    assert (list(root.handlers), root.level) == before, "library touched the root logger"
    top = logging.getLogger("acme_client")
    assert [type(h).__name__ for h in top.handlers] == ["NullHandler"]
    assert top.level == logging.NOTSET, "library set its own level"

Expected Output:

.
1 passed

Common mistakes

basicConfig at import. Error signature: applications' output format changing when the library is upgraded. Root cause: the library configured the root logger. Remediation: remove it; add a NullHandler only.

Routine operation at warning. Error signature: operators adding the library to an ignore list. Root cause: normal events logged as warnings. Remediation: debug for routine, warning only for actionable conditions.

Log and re-raise. Error signature: every failure appearing twice, once without context. Root cause: the library logs exceptions it propagates. Remediation: let the caller log them.

Bodies and tokens in records. Error signature: a security report against the library. Root cause: logging data whose sensitivity the library cannot judge. Remediation: identifiers and metadata only.

A single global logger name. Error signature: applications unable to enable debug for one subsystem. Root cause: every module using the same logger. Remediation: getLogger(__name__) per module.

Frequently Asked Questions