Python Observability

Taming Third-Party Library Loggers in Python

A production service's log volume is mostly not its own code. urllib3 announcing every retry, botocore narrating credential lookups, sqlalchemy.engine echoing statements — all of it arrives through loggers you did not create and cannot edit. This page covers finding those loggers, setting levels where they take effect, and removing the handlers libraries install behind your back. It builds on Python standard library vs third party logging, part of the modern Python logging libraries deep dive section.

Why setting the package level does not always work A logger hierarchy under the root logger. The package logger, boto3, sits below root, and beneath it are several sub-loggers created by different modules of the same library: a client logger, a transport logger, a retry logger and a credentials logger. Setting the package logger to WARNING propagates down to any sub-logger that has no explicit level of its own, because effective level is resolved by walking up until an explicit level is found. But one sub-logger has had its own level set by the library at import time, and that explicit value stops the walk, so it keeps emitting at INFO regardless of what the parent says. The practical consequence is that quietening a package sometimes works completely, sometimes partially, and the difference is invisible from the outside — which is why the reliable approach is to enumerate the loggers that actually exist at runtime and set levels on the specific names. logging.getLogger("boto3").setLevel(WARNING) — and one child ignores it root boto3 level = WARNING boto3.client no level of its own effective: WARNING boto3.transport no level of its own effective: WARNING boto3.retry level = INFO, set by the library effective: INFO boto3.credentials no level of its own effective: WARNING effective level is the nearest explicit level walking up — an explicit child level stops the walk, and nothing reports that it did
Three of the four children obey the package setting. The fourth has its own explicit level and there is no signal, anywhere, that your configuration did not reach it.

Prerequisites

Standard library only.

export LOG_LEVEL=INFO
export LOG_DIAGNOSTIC_DUMP=1     # print the resolved logger table at startup

Implementation

Step 1 — Enumerate what actually exists. Guessing logger names from package names is how a "we silenced boto3" change ends up silencing nothing. logging.Logger.manager.loggerDict holds every logger created so far, so exercise the code path first, then read it.

import logging

def logger_table() -> list[tuple[str, str, bool, int]]:
    """(name, effective level, propagates, handler count) for every logger that exists."""
    rows = []
    for name, obj in sorted(logging.Logger.manager.loggerDict.items()):
        if isinstance(obj, logging.PlaceHolder):        # a namespace with no logger of its own
            continue
        rows.append((
            name,
            logging.getLevelName(obj.getEffectiveLevel()),
            obj.propagate,
            len(obj.handlers),
        ))
    return rows

for name, level, propagates, handlers in logger_table():
    print(f"{name:<44}{level:<10}propagate={propagates!s:<6}handlers={handlers}")

Expected Output:

botocore                                    INFO      propagate=True  handlers=0
botocore.credentials                        INFO      propagate=True  handlers=0
botocore.retryhandler                       INFO      propagate=True  handlers=0
sqlalchemy.engine.Engine                    WARNING   propagate=False handlers=1
urllib3.connectionpool                      DEBUG     propagate=True  handlers=0

Two rows there are findings. sqlalchemy.engine.Engine has propagate=False and a handler of its own, so its records never reach your configuration at all. urllib3.connectionpool is at DEBUG, which at any real request rate is the single largest source of volume in the process.

Step 2 — Set levels on the names, not the packages. Where a library sets an explicit level on a child, only setting that child's level works.

NOISY = {
    "urllib3.connectionpool": logging.WARNING,     # one record per connection reuse
    "botocore.credentials":   logging.WARNING,     # narrates every credential lookup
    "botocore.retryhandler":  logging.INFO,        # keep — retries are real signal
    "sqlalchemy.engine":      logging.WARNING,     # every statement at INFO
    "asyncio":                logging.WARNING,     # slow-callback warnings only
    "charset_normalizer":     logging.WARNING,
}

for name, level in NOISY.items():
    logging.getLogger(name).setLevel(level)

Resist the temptation to set them all to WARNING as a block. botocore.retryhandler at INFO tells you an AWS call is being retried, which is exactly the kind of record that explains a latency incident; botocore.credentials at INFO tells you nothing you will ever act on.

Step 3 — Take back the handlers. Some libraries add a handler at import so their output appears regardless of your configuration. Clear those and leave propagation on.

for name in list(logging.Logger.manager.loggerDict):
    log = logging.getLogger(name)
    if log.handlers and not name.startswith(("app", "myservice")):
        log.handlers = []                          # your root handler will receive them
        log.propagate = True                       # …but only if propagation is on

The propagate = True line is the one that fixes sqlalchemy.engine.Engine from the table above. A library that sets propagate = False has decided its records are its business, and the record never reaches your formatter, your redaction filter, or your sink.

Three states a library logger can be in Three configurations a dependency's logger can arrive in, with the consequence of each. In the first, the logger has no handlers and propagation is on: the record travels up to the root logger and is handled once, in your format, by your sinks — this is the state you want and it requires nothing from you. In the second, the library installed a handler at import time and left propagation on: the record is handled twice, once by the library's handler in its own format and once by yours, which is why the same warning appears in two shapes. In the third, the library set propagate to False, usually alongside its own handler: the record never leaves that logger, so it bypasses your formatter, your redaction filter and your sinks entirely, and no amount of root configuration will bring it back. Only the third state is invisible from reading your own configuration. what state did the library leave its logger in? no handler · propagate on library logger your root handler handled once, in your format the state you want — requires nothing own handler · propagate on library logger + its handler your root handler handled twice, two formats fix: clear the library's handlers own handler · propagate off library logger + its handler never reached bypasses your whole pipeline no redaction, no correlation, no sink only the third is invisible from reading your own configuration — which is why the runtime table above is worth printing at startup
The third column is the dangerous one: those records skip your redaction filter and your correlation fields, and nothing in your configuration hints that they exist.

Step 4 — Keep the records that earn their place. Quietening is not the goal; the goal is a log where every line is worth reading. A short allowlist of dependency records to keep is worth writing down.

Logger Level Why keep it
botocore.retryhandler INFO An AWS call retrying explains a latency spike
urllib3.connectionpool WARNING Retries and pool exhaustion, without the reuse chatter
sqlalchemy.pool WARNING Pool exhaustion is a real incident signal
celery.app.trace INFO Task outcome records — the worker's access log
asyncio WARNING Slow-callback warnings identify event-loop stalls

Step 5 — Make a dependency upgrade visible. A new version can rename a logger, and the symptom is silence, which nothing alerts on. Print the resolved table at startup behind a flag.

import os

if os.environ.get("LOG_DIAGNOSTIC_DUMP") == "1":
    for name, level, propagates, handlers in logger_table():
        logging.getLogger("startup").info(
            "logger configured",
            extra={"logger_name": name, "level": level,
                   "propagate": propagates, "handlers": handlers},
        )
What the stream is actually made of Two stacked bars showing the composition of one service's log volume. Before tuning, the largest share by far is urllib3 connection-pool chatter at debug level, followed by botocore credential lookups, SQLAlchemy statement echoes, and a small remainder of application records — so the great majority of stored, indexed and billed volume is produced by code nobody on the team wrote, and the records an engineer actually wants are a thin slice at the end. After tuning, the same three dependency sources are reduced to their warning-level records only, the deliberately retained signals such as retry handling are kept, and application records become the majority of the stream. The total shrinks by roughly an order of magnitude while the number of useful records goes up, because the retained dependency records are now the ones that indicate something. one service's log volume, by who produced it before urllib3.connectionpool · DEBUG sqlalchemy botocore the last 50 pixels are your application — everything else is code nobody on the team wrote after your application dependency records that survive are the ones that mean something: retries, pool exhaustion, task outcomes the counter-intuitive part total volume falls by roughly an order of magnitude and the number of useful records goes up — because what remains indicates something
The bottom bar is not just shorter. Every dependency record left in it is one that means something happened, which is what makes the stream readable again.

Configuration options

Option Scope Default Recommended
setLevel on sub-logger one namespace inherited set the specific noisy name
handlers = [] library logger library-installed clear, then propagate
propagate library logger usually True force True
disable_existing_loggers dictConfig True False, always
logging.disable(level) global NOTSET avoid — it silences your code too
Startup dump diagnostic off on in staging, behind a flag

Verification

LOG_DIAGNOSTIC_DUMP=1 python -c "import boto3, urllib3, sqlalchemy; import app; app.configure()"

Expected Output:

{"message": "logger configured", "logger_name": "urllib3.connectionpool", "level": "WARNING", "propagate": true, "handlers": 0}
{"message": "logger configured", "logger_name": "botocore.retryhandler", "level": "INFO", "propagate": true, "handlers": 0}
{"message": "logger configured", "logger_name": "sqlalchemy.engine.Engine", "level": "WARNING", "propagate": true, "handlers": 0}

Every row should show handlers=0 and propagate=true for third-party names — that combination is what guarantees the record reaches your pipeline exactly once. Lock it in:

def test_no_library_logger_bypasses_our_pipeline():
    configure()
    for name, obj in logging.Logger.manager.loggerDict.items():
        if isinstance(obj, logging.PlaceHolder) or name.startswith("app"):
            continue
        assert obj.propagate, f"{name} does not propagate — its records never reach us"
        assert not obj.handlers, f"{name} has its own handler — duplicates"

Common mistakes

The level was set on a logger that does not exist yet

Error signature: the configuration is applied at startup and the library is still loud. Root cause: the level was set on a name the library never uses, so a fresh logger was created with that name and nothing else ever touched it. Remediation: enumerate loggerDict after exercising the code path, and set levels on the names that appear there.

Records bypass redaction entirely

Error signature: a dependency logs a connection string in full, although the redaction filter is installed on the root logger. Root cause: the library logger has propagate = False, so the record never reaches the root. Remediation: force propagate = True on library loggers and clear their handlers. The placement rules are in redacting sensitive data in log records.

disable_existing_loggers silenced everything

Error signature: all third-party output disappears the moment dictConfig is applied. Root cause: the flag defaults to True and switches off every logger created before the config ran — which is every library imported at module scope. Remediation: set it to False.

Keeping the configuration honest over time

A logger table tuned today drifts, because dependencies change their logger names, add new ones, and occasionally change what they log at which level. Three habits keep the drift visible rather than silent.

Pin the levels in the declarative configuration. Setting them in code, scattered across modules, means nobody can see the whole policy at once. A loggers block in dictConfig lists every decision in one place, which is also the place a reviewer looks when a dependency upgrade lands.

"loggers": {
    "urllib3.connectionpool": {"level": "WARNING", "propagate": True},
    "botocore.credentials":   {"level": "WARNING", "propagate": True},
    "botocore.retryhandler":  {"level": "INFO",    "propagate": True},
    "sqlalchemy.engine":      {"level": "WARNING", "propagate": True},
    "sqlalchemy.pool":        {"level": "WARNING", "propagate": True},
    "asyncio":                {"level": "WARNING", "propagate": True},
},

Alert on volume by logger, not just in total. A dependency upgrade that starts logging something new at INFO shows up as a step change in that logger's share of the stream, and nowhere else. A simple panel of record count grouped by logger name, looked at once a month, finds these in a minute. It also finds the opposite case, which is quieter and worse: a logger that used to produce records and stopped, because its name changed and your configuration now silences a namespace that no longer exists.

Assert the invariants rather than the values. The specific level for a specific logger is a judgement call that changes. What does not change is that no third-party logger should have handlers of its own or propagation disabled, because either one routes records around your pipeline. That is a property worth testing, and it holds regardless of which levels you chose.

Check Frequency What it catches
Levels in dictConfig at review policy scattered across modules
Volume by logger name monthly a dependency that started or stopped logging
No handlers, propagation on in CI records bypassing your filters and formatters
Startup dump in staging per release a renamed logger, silently silenced

The libraries that need special handling

A few common dependencies have behaviour worth knowing before you meet it.

SQLAlchemy logs SQL through sqlalchemy.engine and has its own echo flag that bypasses level configuration entirely; setting echo=True on the engine writes statements regardless of what the logger says. Leave echo off and control it through the logger. Its sqlalchemy.pool logger is genuinely worth keeping at WARNING, because pool exhaustion is an incident-level signal.

botocore narrates credential resolution at DEBUG and retries at INFO. The retry records are useful — an AWS call being retried explains a latency spike — and the credential ones are pure volume.

urllib3 logs a record per connection creation and reuse at DEBUG, which at any real request rate is the single largest source of volume in a service that makes HTTP calls.

asyncio logs slow-callback warnings at WARNING when debug mode is enabled, which is exactly the signal described in measuring asyncio event loop lag — worth keeping, and worth remembering that it only appears with debug mode on.

Third-party HTTP clients in general tend to log request URLs, which frequently contain tokens in query strings. That is a redaction concern rather than a volume one, and it is one of the strongest arguments for the filter being on the root logger where those records converge.

Frequently Asked Questions

Why is a library still logging after I set its level?

Almost always because you set the level on the wrong logger name. A package called foo often logs under foo.client, foo.transport and foo.retry, and setting foo to WARNING does quieten all three — but only if they have no explicit level of their own. A sub-logger with its own level set by the library ignores the parent entirely, which is why enumerating the real names beats guessing.

Should I use a NullHandler on library loggers?

No — that is advice aimed at library authors, not at applications. A library adds a NullHandler to its own top-level logger so that importing it does not produce a lastResort warning in an application with no logging configured. In an application you want the opposite: no handler on the library logger and propagation on, so its records reach yours.

How do I find out what a library logs before it happens?

Turn everything on in a scratch process — logging.basicConfig(level=logging.DEBUG) — exercise the code path, and read what appears. Then enumerate logging.Logger.manager.loggerDict to see the names that were actually created. Reading a package's source for getLogger calls works too but misses loggers created dynamically.

What about libraries that use warnings instead of logging?

Those never touch the logging module at all. Call logging.captureWarnings(True) and set a warnings filter, which routes them to the py.warnings logger and into the same pipeline. That is a separate mechanism with separate defaults, covered in the unhandled-exceptions guide.

Related Guides