Best Practices for Log Rotation in Python

Reliable log rotation keeps a long-running Python service from exhausting its disk, losing records during rollover, or stalling request threads on rename and metadata work. This walkthrough is for backend engineers and SREs who already write to a file sink and now need that sink to survive weeks of production traffic and several worker processes. It sits inside the handler architecture guide, part of the Python Logging Fundamentals and Structured Data reference.

Prerequisites

Rotation needs no third-party packages — logging.handlers ships with CPython — but the fcntl locking below is POSIX-only (Linux and macOS). Pin the interpreter rather than a library:

# Standard library only. Verified on CPython 3.10-3.13; 3.12.x used throughout.
python --version   # Python 3.12.x
python -c "import fcntl, logging.handlers; print('rotation prerequisites OK')"

Expected Output:

Python 3.12.7
rotation prerequisites OK

Set a writable log directory and an explicit umask so log-shipping agents can read the archives:

export APP_LOG_DIR="/var/log/app"
umask 0022

The queue pattern used in Step 3 is covered in depth in non-blocking logging with QueueHandler, and the JSON formatter shown here follows the conventions in structured logging with the Python standard library.

Implementation

Step 1 — Choose the handler by constraint. Use RotatingFileHandler when disk capacity is the binding limit; it caps each file at maxBytes and keeps backupCount archives. Use TimedRotatingFileHandler when retention is defined in time, setting when="midnight" and interval=1 for daily audit windows. Use WatchedFileHandler when the operating system's rotator owns the trigger. Time-based rotation depends on the system clock and only rolls over on the next emit, so an idle process can defer a rollover and then spike disk usage when traffic resumes.

Step 2 — Serialize rollover with an advisory lock. The stock RotatingFileHandler has no multi-process safety: concurrent doRollover() calls interleave lines and can raise OSError: [Errno 11] Resource temporarily unavailable. Subclass it to take an exclusive fcntl.flock while the stream is open and release it before the rename so the rollover is atomic per worker.

Step 3 — Keep rollover off the request path. Rollover performs file renames and metadata updates that are slow under load. Place the rotating handler behind a QueueHandler/QueueListener so the request thread only enqueues and the background listener absorbs the rollover latency.

import fcntl
import json
import logging
import queue
from logging.handlers import RotatingFileHandler, QueueHandler, QueueListener

log_queue: queue.Queue = queue.Queue(maxsize=10000)  # bounded to cap memory


class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
            "trace_id": getattr(record, "trace_id", None),
            "span_id": getattr(record, "span_id", None),
        }
        return json.dumps(payload, separators=(",", ":"))


class SafeRotatingFileHandler(RotatingFileHandler):
    """RotatingFileHandler that holds an exclusive flock during writes."""

    def _open(self):
        stream = super()._open()
        fcntl.flock(stream.fileno(), fcntl.LOCK_EX)  # block until exclusive
        return stream

    def doRollover(self):
        if self.stream:
            fcntl.flock(self.stream.fileno(), fcntl.LOCK_UN)  # release before rename
            self.stream.close()
            self.stream = None
        super().doRollover()  # rename + shift backups, then reopen (re-locks)


def setup_production_logger(log_path: str = "/var/log/app/service.log"):
    handler = SafeRotatingFileHandler(
        filename=log_path,
        maxBytes=50 * 1024 * 1024,  # 50 MiB per file
        backupCount=5,              # keep 5 archives, ~300 MiB ceiling
        encoding="utf-8",
        delay=True,                 # defer open until first emit
    )
    handler.setFormatter(JSONFormatter())

    listener = QueueListener(log_queue, handler, respect_handler_level=True)
    listener.start()

    logger = logging.getLogger("app")
    logger.setLevel(logging.INFO)
    logger.addHandler(QueueHandler(log_queue))
    logger.propagate = False
    return logger, listener


if __name__ == "__main__":
    logger, listener = setup_production_logger()
    logger.info(
        "Payment processed",
        extra={"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
               "span_id": "00f067aa0ba902b7"},
    )
    listener.stop()  # flush queue and release the lock on shutdown

Expected Output: one JSON object per line in the active file, with the trace context carried through from extra.

{"timestamp":"2026-07-25 10:00:00,123","level":"INFO","message":"Payment processed","module":"__main__","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}

Step 4 — Wire it up declaratively once it works. Prototype in code, then move the final shape into a dictionary configuration so the rotation parameters live beside the rest of your logging setup; see configuring logging with dictConfig for the schema and the () custom-factory key that a handler subclass such as SafeRotatingFileHandler needs.

Step 5 — Validate under load. Drive enough volume to force several rollovers, then confirm the numbered backups appear in sequence and no line is truncated or interleaved. The verification section below gives the exact checks.

How the rotation handlers work internally

RotatingFileHandler and TimedRotatingFileHandler share a parent (BaseRotatingHandler) and differ only in when they decide to roll. On every emit, the handler calls shouldRollover(record). For the size-based handler that check formats the record, seeks to the end of the stream, and compares stream.tell() + len(message) against maxBytes; if the next write would cross the limit, it calls doRollover() first. doRollover() closes the current stream, then renames service.log to service.log.1, shifting service.log.1 to .2 and so on up to backupCount, deleting anything past it, and finally reopens a fresh empty service.log. The rename cascade is backupCount filesystem operations, which is why rollover is the expensive moment and why it must stay off the request path.

TimedRotatingFileHandler computes a rolloverAt timestamp at construction from when and interval, and shouldRollover simply tests time.time() >= rolloverAt. There are two consequences engineers underestimate. First, rotation is emit-driven: the file rolls only when a record arrives after the deadline, so a process that goes idle over a boundary defers its rollover and can then write a single oversized file when traffic resumes. Second, the handler tracks time, not size, so a traffic spike inside one interval can produce a backup far larger than any size cap — pair backupCount with monitoring rather than assuming time-based rotation bounds file size. Both handlers also support utc=True (timed) so boundaries do not drift with daylight-saving changes, and both honor delay=True to defer the initial open until the first emit.

One subtlety worth knowing before you tune levels: shouldRollover runs inside emit, which means a chatty DEBUG logger performs the size check on every suppressed-looking record that still reaches the handler. Keep handler levels tight — the severity contract described in how to configure Python logging for production is also a rotation-cost decision, not only a noise decision.

Multi-process rotation hazards

Neither stock handler is safe across processes. When several workers each hold an open descriptor to service.log and one calls doRollover(), it renames the file out from under the others; the workers that did not rotate keep writing to the now-renamed inode, so their records land in service.log.1 while the rotating worker writes a fresh service.log. Concurrent doRollover() calls can also collide on the rename cascade and raise FileExistsError or, with the advisory lock contended, OSError: [Errno 11] Resource temporarily unavailable. The flock-based subclass above serializes writes and rollover within a single host, which is the right fix when one machine runs several workers. When workers must instead funnel every record to one writer, the single-writer model in thread-safe logging in multiprocessing removes the contention entirely by leaving only one process holding the file.

When the operating system owns rotation instead, use WatchedFileHandler. It never rotates; on every emit it stats the path and compares the current inode and device against the descriptor it holds. If an external rotator (logrotate, journald) has moved or recreated the file, the inode changes, the handler reopens the path, and writing continues into the new file with no lost records — exactly the failure mode that copytruncate cannot avoid. The pairing is: logrotate with create and a postrotate SIGHUP, plus WatchedFileHandler on the Python side, so rotation and reopening are cleanly separated across the process boundary.

import logging
from logging.handlers import WatchedFileHandler

# Python writes; logrotate rotates; WatchedFileHandler reopens on inode change.
handler = WatchedFileHandler("/var/log/app/service.log", encoding="utf-8")
handler.setFormatter(JSONFormatter())
logging.getLogger("app").addHandler(handler)

Expected Output:

# After `logrotate --force`, writes continue into the new inode with no gap:
service.log        # new, active (current inode)
service.log.1.gz   # rotated by logrotate

If your deployment target is a container rather than a VM, the cheapest correct answer is often no rotation at all: write JSON to stdout, let the runtime rotate, and reserve file rotation for hosts you actually own. Third-party libraries take the opposite bet — Loguru builds rotation and retention into the sink itself, as covered in Loguru configuration and sinks — which is worth comparing before you maintain a handler subclass of your own.

Disk-space and retention math

Sizing rotation is arithmetic, not guesswork. With RotatingFileHandler, the worst-case footprint per process is maxBytes * (backupCount + 1) — the active file plus its archives. The configuration above (maxBytes=50 MiB, backupCount=5) caps one process at roughly 300 MiB; an eight-worker host therefore needs about 2.4 GiB of headroom for that one logger, before compression. If logrotate gzips archives, assume a 6–10x reduction on JSON logs and budget for the brief window where both the uncompressed and compressed copies exist during postrotate.

For TimedRotatingFileHandler, size is a function of throughput rather than a hard cap. Estimate average bytes per record (a structured JSON line carrying trace context, as produced when adding trace IDs to log records, is commonly 300–600 bytes), multiply by records per second and the interval length, and that is the expected daily file. The formula daily_bytes = rate * avg_record_bytes * 86400 turns an SLO ("retain 14 days") into a concrete reservation: with 200 records/second at 400 bytes, a day is roughly 6.4 GiB and a 14-day backupCount reserves about 90 GiB uncompressed. Always set backupCount to a finite number; the default of 0 keeps every archive forever and is the most common cause of a disk filling silently weeks after deploy.

Configuration options

Verification

Force a rollover with a tight write loop and inspect the resulting files:

ls -1 /var/log/app/

Expected Output:

service.log
service.log.1
service.log.2

Every line in the active file must still parse as JSON after a rollover — a truncated final line is the signature of an unsynchronized writer:

python -c "import sys,json; [json.loads(l) for l in open('/var/log/app/service.log')]; print('all lines valid JSON')"

Expected Output:

all lines valid JSON

Confirm no file-descriptor leak after repeated rollovers; the count should stay flat across dozens of rolls:

ls /proc/self/fd | wc -l

Finally, turn on the logging module's own diagnostics (logging.raiseExceptions = True, the default) during the load test so a failed rename surfaces on stderr instead of being swallowed.

Common mistakes

Using copytruncate with OS logrotate on a Python process Error signature: the log file is non-empty on disk but a tail -f shows a gap of several seconds after each rotation, and file offsets in lsof are far larger than the file size. Root cause: Python keeps the original descriptor open; copytruncate copies then truncates in place, so the process keeps writing at its old offset into a sparse hole. Remediation: configure logrotate with create plus a postrotate SIGHUP, and use WatchedFileHandler (or reopen the handler on that signal) so the descriptor follows the new inode.

Synchronous rotation blocking the request thread Error signature: p99 latency spikes by tens of milliseconds at a regular interval that matches the rollover cadence, with no matching change in downstream dependencies. Root cause: doRollover performs backupCount renames plus a reopen inline on the calling thread. Remediation: front the rotating handler with a QueueHandler and let a background QueueListener absorb the cost, exactly as in Step 3.

Assuming os.rename is atomic on network filesystems Error signature: duplicated or zero-length archives such as service.log.2 appearing twice in different states, or OSError: [Errno 116] Stale file handle during rollover. Root cause: rollover relies on atomic renames, and NFS or EFS mounts do not guarantee that under concurrent access. Remediation: keep logs on a local volume, or write locally and ship from there with an agent.

Leaving backupCount at its default of zero Error signature: the incident timeline you need is missing — the file contains only minutes of history, or conversely the disk fills weeks after a clean deploy. Root cause: with backupCount=0 the size handler keeps no archives at all and truncates on roll, while an unset backupCount on a timed handler retains every archive forever. Remediation: derive a finite value from the retention math above so the storage ceiling is a decision rather than an accident.

Related

• Handler architecture — the parent guide covering handler topology, levels and backpressure across sinks.
• Non-blocking logging with QueueHandler — the queue layer that keeps the rollover cost off your request threads.
• Thread-safe logging in multiprocessing — the single-writer alternative to per-worker file locking.
• Configuring logging with dictConfig — how to declare the rotating handler, its formatter and its levels in one config block.
• Structured logging with the Python standard library — the JSON record shape whose size drives the retention arithmetic above.

Frequently Asked Questions