Configuring Python Logging with dictConfig: One YAML File, Two Destinations

You need a single logging setup that an operator can edit without touching Python: readable lines on the console, filtered JSON in a rotating file, and a different level for a chatty module — all applied by one call at startup. This page is for backend engineers and SREs wiring a service for the first time, or replacing a pile of imperative addHandler calls. It is the concrete companion to the logging configuration and dictConfig guide and part of the Python Logging Fundamentals and Structured Data guide; the sections below build one complete logging.yaml and the twelve-line bootstrap that applies it.

logging.config.dictConfig consumes a plain dictionary that describes the entire logging tree at once: formatters, filters, handlers, and loggers, plus a small set of top-level switches. Building that dictionary in YAML rather than in Python keeps operational knobs — levels, file paths, rotation sizes — out of code and lets you ship a different file per environment. The schema is versioned by the mandatory version: 1 key; that anchor is what tells dictConfig to use the dictionary schema rather than the older fileConfig INI format, and it is the only value the schema currently accepts.

Prerequisites

# YAML parsing only; json, logging and logging.config are stdlib.
pip install "pyyaml>=6.0,<7.0"

Python 3.8 or newer is enough for everything below; the optional queue wiring noted in the configuration table needs 3.12. Two environment variables drive the overrides used later:

export LOG_LEVEL=DEBUG          # falls back to the YAML value when unset
export LOG_FILE=/var/log/app/app.log   # falls back to ./app.log when unset

Implementation

The dictionary schema has five sections you will fill in — formatters, filters, handlers, loggers, and the root logger — sitting under three top-level switches (version, disable_existing_loggers, and optionally incremental). dictConfig resolves them in dependency order: formatters and filters first, then handlers that reference them by name, then loggers that attach handlers. A typo in a name therefore surfaces as a ValueError at startup rather than a silent miswire discovered days later in production.

Step 1 — Write a custom JSON formatter via the () factory. The stdlib Formatter interpolates a format string but does not emit valid JSON when messages contain quotes or newlines. A tiny json.dumps-based formatter is safer and needs no third-party package; the same technique is developed further in structured logging with the Python standard library. The () key tells dictConfig to treat the value as a factory: it imports the dotted path and calls it, passing any sibling keys as keyword arguments. That means your formatter accepts configuration straight from YAML. Place it in an importable module, here app/log_json.py.

# app/log_json.py
import json
import logging


class JsonFormatter(logging.Formatter):
    """Serialize each record as a single JSON object."""

    def __init__(self, fields: list[str] | None = None) -> None:
        # fields arrives from the YAML sibling keys via the () factory
        super().__init__()
        self.fields = fields or ["ts", "level", "logger", "msg"]

    def format(self, record: logging.LogRecord) -> str:
        base = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S"),
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
        }
        payload = {k: base[k] for k in self.fields if k in base}
        if record.exc_info:                       # attach traceback if present
            payload["exc"] = self.formatException(record.exc_info)
        return json.dumps(payload, ensure_ascii=False)

Step 2 — Write a filter to drop noisy records. A filter is any callable, or any object with a filter(record) method; returning falsy discards the record before the handler emits it. Filters are configured through the same () factory, and a handler attaches them by name. Here a filter suppresses health-check lines that would otherwise dominate the file and inflate storage cost.

# app/log_filters.py
import logging


class DropHealthChecks(logging.Filter):
    """Discard records whose message mentions the health endpoint."""

    def filter(self, record: logging.LogRecord) -> bool:
        # returning False drops the record; True lets it through
        return "/healthz" not in record.getMessage()

Step 3 — Declare the configuration in YAML. The file names two formatters, one filter, two handlers, and two module loggers. The custom formatter and filter are wired with the () key; the stream handler resolves sys.stdout through the ext:// prefix, which tells dictConfig to import a dotted name rather than treat the string literally. propagate: false on each named logger prevents records from also reaching the root, and the rotation settings follow the sizing rules in log rotation best practices.

# logging.yaml
version: 1
disable_existing_loggers: false

formatters:
  console:
    format: "%(asctime)s %(levelname)-8s %(name)s | %(message)s"
    datefmt: "%Y-%m-%dT%H:%M:%S"
  json:
    "()": app.log_json.JsonFormatter      # custom class, instantiated by dictConfig
    fields: [ts, level, logger, msg]      # passed as a kwarg to __init__

filters:
  no_health:
    "()": app.log_filters.DropHealthChecks

handlers:
  stdout:
    class: logging.StreamHandler
    level: DEBUG
    formatter: console
    stream: ext://sys.stdout
  file:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: json
    filters: [no_health]                   # attach the filter by name
    filename: app.log
    maxBytes: 10485760                     # 10 MiB per file
    backupCount: 5                         # keep five rolled files

loggers:
  app.api:
    level: DEBUG
    handlers: [stdout, file]
    propagate: false
  app.db:
    level: WARNING                         # quieter than the rest of the app
    handlers: [stdout, file]
    propagate: false

root:
  level: WARNING
  handlers: [stdout]

Step 4 — Load, overlay the environment, and apply it once. Read the file, deserialize with yaml.safe_load (never yaml.load, which can instantiate arbitrary objects from untrusted input), apply environment overrides to the plain dictionary, then call dictConfig before importing any module that logs. Doing the load in code — rather than through a framework setting such as Django's LOGGING — keeps the path explicit and unit-testable, and the overlay is what lets one file serve every environment.

# app/bootstrap.py
import logging.config
import os
from pathlib import Path

import yaml  # pyyaml>=6.0,<7.0


def configure_logging(path: str = "logging.yaml") -> None:
    config = yaml.safe_load(Path(path).read_text(encoding="utf-8"))

    # Overrides are applied to the plain dict, before any object is built.
    level = os.environ.get("LOG_LEVEL")
    if level:
        config["loggers"]["app.api"]["level"] = level.upper()

    log_file = os.environ.get("LOG_FILE")
    if log_file:
        Path(log_file).parent.mkdir(parents=True, exist_ok=True)  # handler will not create it
        config["handlers"]["file"]["filename"] = log_file

    logging.config.dictConfig(config)      # raises ValueError on any bad reference

Step 5 — Emit from two modules. Each module fetches its own named logger. Because the names match the YAML keys, app.api runs at DEBUG and app.db is held at WARNING. Dotted names matter here: a logger named app.db.pool with no explicit entry inherits the app.db level through the hierarchy, so one key configures a whole subtree — the mechanism explained in log levels and severity mapping.

# app/main.py
from app.bootstrap import configure_logging

configure_logging()                  # MUST run before the loggers are used

import logging

api_log = logging.getLogger("app.api")
db_log = logging.getLogger("app.db")

api_log.debug("request received: GET /orders/42")
api_log.info("order 42 served in 12ms")
api_log.info("GET /healthz ok")                    # dropped by the file filter
db_log.debug("SELECT * FROM orders WHERE id=42")   # suppressed at WARNING
db_log.warning("connection pool at 90% capacity")

incremental vs disable_existing_loggers

Two top-level switches change how dictConfig treats loggers that already exist when it runs, and both cause silent surprises when left at their defaults.

disable_existing_loggers (default true) disables every logger created before the call that is not explicitly named in the new configuration. Libraries grab their loggers at import time, so the default silently mutes your dependencies — the single most common reason a service stops showing urllib3 or SQLAlchemy warnings after someone "fixed" logging. Setting it to false leaves those loggers enabled and is almost always what an application wants; note that it disables rather than removes them, so a logger muted this way still exists and can be re-enabled by a later full configuration.

incremental (default false) changes the semantics wholesale: when true, dictConfig ignores formatters, filters, and handler and logger construction, and only adjusts the level and propagate of objects that already exist. It is meant for tuning a live configuration — raising one logger to DEBUG from an admin endpoint, for instance — without tearing down handlers, which would close open file descriptors and lose buffered records. You cannot add a handler or formatter incrementally; the attempt is skipped without an error. The practical pattern is a full, non-incremental config at startup and incremental updates only for runtime level changes.

Configuration options

Two prefixes are worth remembering: ext:// resolves a dotted name outside the configuration (ext://sys.stderr, ext://app.sinks.socket), while cfg:// resolves a path inside the same dictionary (cfg://handlers.file.filename), which lets one value be defined once and referenced elsewhere.

Verification

Run python -m app.main with the defaults. The console, using the console formatter, shows readable lines:

Expected Output (stdout):

2026-06-19T12:11:03 DEBUG    app.api | request received: GET /orders/42
2026-06-19T12:11:03 INFO     app.api | order 42 served in 12ms
2026-06-19T12:11:03 INFO     app.api | GET /healthz ok
2026-06-19T12:11:03 WARNING  app.db | connection pool at 90% capacity

The app.db DEBUG line is absent because that logger is pinned to WARNING. The rotating app.log simultaneously receives JSON records. The file handler is at INFO, so app.api's DEBUG line is excluded there, and the no_health filter strips the /healthz line that did reach the console:

Expected Output (app.log):

{"ts": "2026-06-19T12:11:03", "level": "INFO", "logger": "app.api", "msg": "order 42 served in 12ms"}
{"ts": "2026-06-19T12:11:03", "level": "WARNING", "logger": "app.db", "msg": "connection pool at 90% capacity"}

Each record appears once per destination, confirming propagate: false keeps the root logger from re-emitting it, and the health-check line is present on the console but absent from the file, confirming the filter is attached to the file handler only.

Assert the wiring in CI rather than trusting a manual read of the output. The configuration is just a dictionary, so a test can apply it and inspect the resulting objects:

# tests/test_logging_config.py
import logging
import logging.config

import pytest
import yaml  # pyyaml>=6.0,<7.0


@pytest.fixture(autouse=True)
def _config(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)              # keep app.log out of the repo
    config = yaml.safe_load(open("logging.yaml", encoding="utf-8").read())
    logging.config.dictConfig(config)


def test_levels_and_handlers():
    api = logging.getLogger("app.api")
    assert api.level == logging.DEBUG        # per-module level from YAML
    assert api.propagate is False            # no duplicate via root
    assert {h.name for h in api.handlers} == {"stdout", "file"}


def test_health_filter_only_on_file(caplog):
    file_handler = next(h for h in logging.getLogger("app.api").handlers
                        if h.name == "file")
    record = logging.LogRecord("app.api", logging.INFO, __file__, 1,
                               "GET /healthz ok", None, None)
    assert file_handler.filter(record) is False   # dropped before the file

Expected Output (pytest):

tests/test_logging_config.py::test_levels_and_handlers PASSED
tests/test_logging_config.py::test_health_filter_only_on_file PASSED

Common mistakes

• Error signature: ValueError: Unable to configure formatter 'json' at startup, with a chained ModuleNotFoundError: No module named 'app.log_json'. Root cause: the dotted path under () does not resolve — usually the process was started from a directory where the package is not on sys.path, or the module was renamed after the YAML was written. Remediation: check the path independently with python -c "import app.log_json" from the same working directory the service uses, and keep the config test above in CI so a rename fails the build rather than the deploy.

• Error signature: ValueError: Unable to configure handler 'file' with a chained FileNotFoundError: [Errno 2] No such file or directory: '/var/log/app/app.log'. Root cause: file handlers create the file but not its parent directory, so a container or fresh host without the log directory aborts the whole configuration — including the console handler, leaving you with no logs at all to debug it. Remediation: create the parent directory in the bootstrap (as configure_logging does above) or provision it in the image, and prefer a path under a writable volume rather than one owned by root.

• Error signature: every line appears twice on the console, once formatted by console and once by the default root format. Root cause: a named logger keeps propagate at its default of true while the root also has handlers attached, so each record is emitted by the module handlers and again by the root's. Remediation: attach handlers in exactly one place per branch of the tree — set propagate: false on named loggers that carry their own handlers, or leave them handler-free and let the root do all the emitting.

• Error signature: dependency logs (urllib3, sqlalchemy.engine, a vendored client) go completely silent after logging is configured, while your own modules log normally. Root cause: disable_existing_loggers was left at its default true, so every logger created at import time — before configure_logging ran — was disabled. Remediation: set disable_existing_loggers: false, and if you genuinely want a chatty dependency quiet, name it explicitly in loggers with level: WARNING instead of muting it by accident.

• Error signature: a newly added handler never writes anything, and no error is raised. Root cause: incremental: true was left in the file from an earlier runtime tweak, so dictConfig skipped the handler construction entirely and only touched levels on existing objects. Remediation: use a full, non-incremental configuration whenever objects must be created, and keep incremental dictionaries in a separate, minimal payload used only by the endpoint that adjusts levels.

• Error signature: the first few records of every boot use the default WARNING:root: format, and DEBUG lines from imported modules are missing. Root cause: a module that logs at import time was imported before configure_logging ran, so those records went through the last-resort handler. Remediation: make the bootstrap call the first statement in the entry point, before application imports, exactly as app/main.py does above; in frameworks with an application factory, call it at the top of the factory.

Related

• Logging configuration and dictConfig — the parent guide covering the schema, incremental updates, and framework integration.
• Structured logging with the Python standard library — a production-grade version of the JSON formatter wired in above.
• Non-blocking logging with QueueHandler — move file and network I/O off the request thread once this config is in place.
• Log rotation best practices in Python — how to size maxBytes and backupCount for real retention targets.
• How to configure Python logging for production — the level policy and environment overrides this file plugs into.

Frequently Asked Questions