Loguru Rotation, Retention and Compression

Loguru's file sink is the reason many teams pick it: rotation, retention and compression are three keyword arguments rather than three classes. This page covers what those arguments actually do, the timing behaviour that surprises people, and the deployment shape where in-process rotation stops being safe. It builds on Loguru configuration and sinks, part of the modern Python logging libraries deep dive section.

Prerequisites

pip install "loguru>=0.7.2,<1.0.0"

export LOG_DIR=/var/log/checkout
export LOG_ROTATION="100 MB"
export LOG_RETENTION="14 days"

Implementation

Step 1 — Configure one sink with all three settings. logger.add() returns a sink ID; logger.remove() with no argument clears the default stderr sink, which is almost always the first line of a production setup.

import os
import sys
from loguru import logger

logger.remove()                                   # drop the default stderr sink

logger.add(
    f"{os.environ['LOG_DIR']}/app.log",
    rotation=os.environ.get("LOG_ROTATION", "100 MB"),
    retention=os.environ.get("LOG_RETENTION", "14 days"),
    compression="gz",
    enqueue=True,                                 # writes and rotation on a dedicated thread
    serialize=True,                               # one JSON object per line
    level="INFO",
    backtrace=False,                              # do not expand frames in production
    diagnose=False,                               # never render variable values
)

diagnose=False is not optional in production. With it enabled, Loguru's exception rendering includes the values of local variables in each frame, which will eventually put a password, a token, or a customer record into a log file.

Step 2 — Choose the rotation trigger deliberately. Loguru accepts three shapes, and each answers a different question.

rotation="100 MB"        # size — predictable disk usage, unpredictable time boundaries
rotation="00:00"         # daily at midnight — predictable boundaries, unpredictable size
rotation="1 week"        # duration since the file was created
rotation=lambda msg, file: file.tell() > 5e8 and msg.record["level"].no >= 40  # custom

The timing detail matters: the condition is checked on write, not on a timer. A "00:00" rotation on a service that receives no traffic between 23:50 and 06:00 rotates at 06:00, and the resulting file is labelled with the time it was rotated. If exact daily boundaries matter — for a compliance export, say — rotate externally with logrotate and let Loguru simply append.

Step 3 — Make retention actually delete. Retention accepts a count, a duration, or a callable. It runs after each rotation and globs the sink's directory.

retention=10             # keep the ten most recent files
retention="14 days"      # delete anything older
retention=lambda files: [os.remove(f) for f in sorted(files)[:-10]]   # explicit

Point the sink at a directory nothing else writes to. Retention selects candidates by pattern-matching the sink's filename, and a shared /var/log directory can contain files that match by accident — a rotated file from a previous deployment, for instance.

Step 4 — Enqueue the sink. enqueue=True moves every write, rotation and compression to a dedicated thread and makes the sink safe across processes that Loguru itself spawned. It is also what keeps a 200 MB gzip from happening on a request thread.

logger.add("/var/log/checkout/app.log", rotation="100 MB", compression="gz", enqueue=True)

The cost is a queue and a thread, and one obligation: call logger.complete() (or logger.remove()) at shutdown so the queue drains before the process exits. The async-specific behaviour of this flag is covered in async logging with Loguru's enqueue.

Step 5 — Stop rotating when there is more than one process. Two gunicorn workers with the same file sink will both decide the file is too big and both rename it. Records written after the first rename go to a file the second worker has already moved.

# under gunicorn, uvicorn --workers, or a Celery prefork pool:
logger.remove()
logger.add(sys.stdout, serialize=True, level="INFO", enqueue=True)   # no files, no rotation

Let the container runtime, systemd, or logrotate own the files. This is not a Loguru limitation — the standard library's RotatingFileHandler has exactly the same problem, and for the same reason.

Configuration options

Verification

from loguru import logger
import os, pathlib

logger.remove()
logger.add("/tmp/rot/app.log", rotation="1 KB", retention=3, compression="gz", enqueue=True)

for i in range(2000):
    logger.info("record {}", i)

logger.complete()                    # drain the queue before inspecting
print(sorted(p.name for p in pathlib.Path("/tmp/rot").iterdir()))

Expected Output:

['app.2026-08-02_13-41-02_884321.log.gz',
 'app.2026-08-02_13-41-02_913776.log.gz',
 'app.2026-08-02_13-41-02_941208.log.gz',
 'app.log']

Three compressed files plus the active one — retention deleted the rest. Two things to check specifically: the count matches retention, and the rotated files are .gz rather than plain, which confirms compression ran rather than silently doing nothing.

Common mistakes

The disk fills anyway

Error signature: rotation is clearly working — dozens of files — and the volume is full. Root cause: rotation was set and retention was not, so nothing is ever deleted. Remediation: always set both. Rotation divides the space; only retention bounds it.

Records go missing under gunicorn

Error signature: a worker's records stop appearing after a rotation, and the file sizes do not add up. Root cause: multiple processes rotating the same path independently. Remediation: log to stdout under multiple processes and let the platform handle files.

A latency spike every time a file rotates

Error signature: a periodic P99 spike whose interval matches the rotation cadence. Root cause: compression runs inline in the writing thread. Remediation: set enqueue=True, which moves rotation and compression to the sink thread.

Passwords in the log file

Error signature: an exception block in the log contains local variable values, one of which is a credential. Root cause: diagnose=True, Loguru's default, renders frame locals in tracebacks. Remediation: set diagnose=False and backtrace=False in every non-development configuration, and pair it with the redaction described in redacting sensitive data in log records.

Files or stdout

Rotation only matters if files are the right destination, and in a containerised deployment they usually are not. The decision is worth making explicitly rather than inheriting from whichever example the project started from.

Stdout is right when the platform collects it: a container runtime with a logging driver, systemd with the journal, a supervisor that pipes to an agent. In that arrangement the process writes to a pipe and something else owns durability, rotation and retention — all of which it does better, because it can do them across every process on the host rather than per process. The application configuration reduces to one sink and no file management at all.

Files are right when there is no collector, when the log has to survive the loss of the network, or when an on-host process reads it — a security agent tailing an audit file, a legacy shipper, a support tool. They are also right in a single-process deployment where the operational simplicity of "the log is at this path" outweighs everything else.

Both is a mistake more often than it looks. Writing to stdout and a file doubles the I/O, doubles the storage, and creates two sources of truth that will eventually disagree about what was retained. If a file is needed for one specific consumer, give that consumer its own sink with its own level and format rather than duplicating the whole stream.

Retention as a policy, not a setting

retention="14 days" is a number in a configuration file, and the number usually comes from nowhere in particular. Three inputs decide it, and they are worth writing down next to the setting.

The first is how far back an investigation realistically reaches. For most services this is days rather than weeks: an incident is investigated while it is fresh, and a question asked a month later is asked of metrics rather than logs. The second is any compliance requirement, which is usually longer and usually applies to a specific subset — audit and access records — rather than to everything. The third is cost, which is what actually constrains the first two.

The productive shape that follows is different retention for different streams rather than one number for everything: a short window for the high-volume operational stream, and a long one for the low-volume audit stream in its own sink. Loguru makes that straightforward, because a sink is a per-destination configuration.

logger.add("/var/log/app/app.log", rotation="100 MB", retention="7 days",
           compression="gz", enqueue=True, level="INFO")

logger.add("/var/log/app/audit.log", rotation="1 day", retention="400 days",
           compression="gz", enqueue=True, level="INFO",
           filter=lambda record: record["extra"].get("audit") is True)

The filter argument is what makes the second sink cheap: it receives only the records explicitly marked as audit events, so its volume is a tiny fraction of the first and a year of retention costs less than a week of the operational stream.

Verifying retention actually runs

Retention executes after a rotation, which means a sink that never rotates never deletes anything. A service quiet enough that its file has not reached the size threshold in a month has a month of records in one file and a retention setting that has never executed. Checking the directory occasionally is the whole test — a file count that is not what the retention setting implies, or an active file far older than the rotation interval suggests, is the symptom.

Related

• Loguru configuration and sinks — the parent guide: sinks, formats, and the handler model.
• Async logging with Loguru's enqueue — what enqueue=True does under asyncio and at shutdown.
• Implementing custom sinks in Loguru — when a file is not the destination.
• Intercepting standard logging with Loguru — getting dependency records into this sink.
• Best practices for log rotation in Python — the same problem with the standard library's handlers.

Frequently Asked Questions