Python Observability

HTTP and Webhook Logging Handlers

A logging handler that sends records over HTTP puts a network call inside the logging path, and the logging path runs on every thread that logs. That is occasionally the right trade — a notification to a chat webhook when something needs a human, a record that must reach a system with no collector — and it is a poor default for general log shipping. This page covers when the trade is worth making, and the queue, batching, timeout and rate limit that make it safe when it is. It is a task article under handler architecture, part of the Python logging fundamentals and structured data section.

Where the network call happens Two arrangements of the same HTTP logging handler during a period when the receiving endpoint is slow. In the first, the handler is attached directly to the logger, so the HTTP request happens on the thread that called the logger. Every request thread that logs an error waits for the connection to time out, thirty seconds by default, and the service's latency rises to match; the service is effectively taken down by the system it was trying to report to. In the second, the logger's handler is a QueueHandler, which only enqueues, and a separate listener thread owns the HTTP handler. Request threads return immediately. The listener thread waits on the slow endpoint, the bounded queue fills, and further records are dropped and counted. The service's latency is unaffected. The note records that the handler is identical in both cases and only its position differs. the same handler, the endpoint slow, two positions attached directly request thread log.error → HTTP POST → waits for the timeout, inside the request every thread that logs is stuck — the reporter takes down the service behind a queue request thread enqueue, return — microseconds listener thread waits on the endpoint · queue fills · excess dropped and counted identical handler — only its position decides whether an outage elsewhere becomes yours
The network call has to happen somewhere. Behind a queue it happens on a thread nobody is waiting for.

Prerequisites

pip install "requests>=2.31.0,<3.0.0"

QueueHandler, QueueListener and HTTPHandler are in the standard library.

Implementation

Step 1 — Decide which records justify a network handler. The strongest case is a small, high-value subset: records that need a human's attention now, sent to a chat or paging webhook. The weakest is general log shipping, which a collector reading standard output does more robustly — it survives network partitions, buffers on disk and never puts the network in the request path, as covered in log shipping and collection. A filter that selects only the records meant for the webhook keeps the handler's scope narrow.

import logging

class NotifyFilter(logging.Filter):
    """Only records explicitly marked for notification reach the webhook."""
    def filter(self, record: logging.LogRecord) -> bool:
        return record.levelno >= logging.ERROR and getattr(record, "notify", False)

Step 2 — Put the handler behind a queue. The request thread must never make the HTTP call. A QueueHandler on the logger enqueues and returns in microseconds; a QueueListener on a background thread owns the HTTP handler and makes the calls. A bounded queue means an unreachable endpoint fills the queue and drops records rather than growing memory. The pattern is developed in non-blocking logging with QueueHandler.

import queue
from logging.handlers import QueueHandler, QueueListener

notify_queue: "queue.Queue[logging.LogRecord]" = queue.Queue(maxsize=1000)
listener = QueueListener(notify_queue, webhook_handler, respect_handler_level=True)
listener.start()

notify_logger = logging.getLogger("notify")
notify_logger.addHandler(QueueHandler(notify_queue))

Step 3 — Batch, with a short timeout. The standard HTTPHandler sends one request per record and has no timeout, so a slow endpoint holds the listener indefinitely. A small subclass that accumulates records and posts them together, with a timeout of a few seconds, bounds the cost of a slow endpoint to one batch.

import json
import sys
import time
import threading
import requests
from prometheus_client import Counter

DELIVERY_FAILURES = Counter("webhook_delivery_failures_total", "Failed webhook posts")

class BatchingWebhookHandler(logging.Handler):
    def __init__(self, url: str, max_batch: int = 20, max_wait: float = 5.0,
                 timeout: float = 3.0):
        super().__init__()
        self.url, self.max_batch, self.max_wait, self.timeout = url, max_batch, max_wait, timeout
        self._buf: list[dict] = []
        self._last = time.monotonic()
        self._lock = threading.Lock()
        self._session = requests.Session()

    def emit(self, record: logging.LogRecord) -> None:
        with self._lock:
            self._buf.append({"level": record.levelname, "logger": record.name,
                              "message": self.format(record)})
            due = len(self._buf) >= self.max_batch or \
                  time.monotonic() - self._last >= self.max_wait
            batch, self._buf = (self._buf, []) if due else ([], self._buf)
        if batch:
            self._post(batch)

    def _post(self, batch: list[dict]) -> None:
        self._last = time.monotonic()
        try:
            self._session.post(self.url, data=json.dumps({"records": batch}),
                               headers={"content-type": "application/json"},
                               timeout=self.timeout)
        except requests.RequestException as exc:
            # 1. Never log this through a logger that routes back here.
            DELIVERY_FAILURES.inc()
            print(f"webhook delivery failed: {exc.__class__.__name__}", file=sys.stderr)

    def flush(self) -> None:
        with self._lock:
            batch, self._buf = self._buf, []
        if batch:
            self._post(batch)

Step 4 — Rate limit and aggregate. During an incident the records worth notifying about are produced in bursts, and a webhook receiving hundreds of identical messages is both useless and likely to throttle the sender. A limit on records per minute, with a summary of how many were suppressed, keeps the channel readable.

class RateLimitFilter(logging.Filter):
    def __init__(self, per_minute: int = 10):
        super().__init__()
        self.per_minute, self._window, self._count, self._suppressed = per_minute, 0, 0, 0

    def filter(self, record: logging.LogRecord) -> bool:
        window = int(time.time() // 60)
        if window != self._window:
            if self._suppressed:
                record.msg = f"{record.msg} (+{self._suppressed} suppressed last minute)"
            self._window, self._count, self._suppressed = window, 0, 0
        self._count += 1
        if self._count > self.per_minute:
            self._suppressed += 1
            return False
        return True

Step 5 — Never report delivery failures through the same path. A handler that logs its own errors through the logging system can send an error record — about the endpoint failing — to the failing endpoint, which fails, which logs another error. Delivery failures belong in a counter and on standard error, never in a logger that routes to the handler that failed.

Expected Output: during an endpoint outage, the service is unaffected and the failures are counted.

webhook delivery failed: ConnectTimeout
webhook_delivery_failures_total 14
notify_queue_dropped_total 0
p99 request latency  211 ms   (unchanged)
Keeping a channel readable during an incident An incident produces four hundred error records marked for notification over two minutes. Without rate limiting, all four hundred are posted to a chat webhook: the channel fills with near-identical messages that nobody can read, the endpoint begins throttling the sender, and later messages — including the one reporting recovery — are rejected. With a limit of ten per minute and aggregation, the first ten records each minute are posted, and the next message carries a note saying how many were suppressed. The channel shows twenty-two messages that together describe the incident's shape and size, and the recovery message arrives. The note records that the aggregated count preserves the information that matters — how bad it was — while the individual records remain available in the ordinary log store. 400 notify-worthy errors in two minutes no limit 400 near-identical messages · sender throttled · recovery message rejected 10/min + summary "+190 suppressed" "+188 suppressed" recovered 22 messages that describe the incident, instead of 400 that bury it the suppressed count keeps the size of the problem visible every individual record still reaches the ordinary log store through the normal path the webhook is for attention, not for storage
A notification channel is for attention. Aggregation keeps the attention on the incident rather than on scrolling past four hundred copies of it.

Why this is rarely the right primary path

It is worth being explicit about why the standard advice is to ship logs through a collector rather than from the application, because the HTTP handler looks simpler and that simplicity is misleading.

The failure modes are inverted. A collector reading standard output keeps working when the network is partitioned, buffers on the node's disk, and retries without the application's involvement. An HTTP handler has none of that: when the endpoint is unreachable, records are lost as soon as the queue fills, and nothing outside the process can recover them.

Credentials live in the application. A webhook URL or an ingestion token must be configured in every service that uses the handler, rotated in every service when it changes, and protected in every service's secret store. A collector holds one copy.

The network path belongs to the request's process. Even behind a queue, the handler's connection pool, TLS handshakes and retries consume resources in the same process that serves requests. At low volume this is invisible; at high volume it competes.

Shutdown loses data. Records in the queue when the process exits are lost unless the listener is stopped and the handler flushed, as described in graceful shutdown and telemetry flush. A collector reading from a file loses nothing on application shutdown.

For a chat notification, none of these matters much: the volume is tiny, losing a notification during a network outage is acceptable, and there is no collector-side equivalent of a formatted alert message. For general log shipping, all of them matter, and the collector wins.

Designing the notification itself

A record that reaches a human through a chat channel is read differently from one in a log store, and a few choices about its content make it far more useful.

Lead with what happened and where. The first line a reader sees should name the service, the environment and the event in plain terms — "checkout (prod): payment provider rejecting all charges" — because notifications are read on phones and in previews where only that line is visible. The formatted log record, with its timestamp and logger name first, is the wrong shape for this audience.

Link to the investigation, not the evidence. Including a trace identifier or a pre-built query link lets the reader go from the notification to the relevant trace or log search in one click. Including the full traceback in the message is usually counterproductive: it is unreadable in a chat client, and it may contain data that should not be broadcast to everyone in a channel.

Say whether action is needed. A notification that is informational and one that requires somebody to act should look different, and the handler is the right place to make that distinction — a field on the record, set by the code that knows, mapped to a mention or a colour by the formatter. Channels where every message looks equally urgent train people to ignore all of them.

These are formatting decisions, so they belong in a dedicated formatter attached to the webhook handler rather than in the log call. The call site keeps logging a structured record; the webhook's formatter turns it into something a person can act on.

Keeping the network off the request path Four components that keep an HTTP logging handler from slowing requests. The application logs normally; a QueueHandler puts the record on an in-memory queue in microseconds. A QueueListener thread takes records off the queue. A filter on the listener passes only records at ERROR or above, or those tagged for notification. The HTTP handler posts them in batches with a short timeout, and on failure counts the drop rather than retrying forever. The note says the request thread never waits on the network, and a slow webhook costs queued records, not latency. log call → queue → listener → filter → post QueueHandler request thread enqueue · microseconds QueueListener background thread drains the queue filter ERROR and above, or tagged records HTTP handler batched posts, short timeout, count drops the request thread never waits on the network a slow webhook costs queued records, not request latency
The only work on the request path is putting a record on a queue. Everything that can be slow happens on another thread.

Configuration options

Setting Value Why
Scope a filtered subset the network is for high-value records only
Position behind QueueHandler request threads never wait on the network
Queue bound ~1 000 records an outage drops rather than grows memory
Batch size 10–20 records fewer requests, bounded payload
Batch wait 5 s latency of notification vs request count
Request timeout 3 s a slow endpoint costs one batch
Rate limit 10 per minute, with summary readable channel, no throttling
Failure reporting counter and stderr never back through the handler

Verification

Stop the endpoint and confirm the service does not notice.

# point the handler at an address that drops connections, then load-test
WEBHOOK_URL=http://10.255.255.1:9 python loadtest.py --rps 200 --duration 60

Expected Output: unchanged latency, and failures counted rather than propagated.

p99 latency            208 ms
webhook failures        12
notify queue dropped     0

A latency increase during this test means the HTTP call is happening on a request thread — the queue is missing or the handler is attached directly as well as behind it.

Common mistakes

The standard HTTPHandler attached directly. Error signature: service latency rising when the log endpoint is slow. Root cause: one synchronous request per record on the calling thread, with no timeout. Remediation: a batching handler behind a queue listener.

Using it for general log shipping. Error signature: logs lost during every network blip. Root cause: no disk buffer and no retry outside the process. Remediation: ship through a collector; keep the HTTP handler for notifications.

No rate limit. Error signature: a flooded channel and a throttled sender during an incident. Root cause: every notify-worthy record posted. Remediation: limit and aggregate.

Logging delivery failures through logging. Error signature: a feedback loop of error records aimed at a failing endpoint. Root cause: the handler's error path routes back to itself. Remediation: a counter and standard error.

No flush at shutdown. Error signature: the last notifications before a crash never arrive. Root cause: records left in the queue and the batch buffer. Remediation: stop the listener and flush the handler on termination.

Frequently Asked Questions

Should a Python service ship its logs over HTTP directly?

Generally not as its primary path. Writing to standard output and letting a collector ship is more robust, because it survives network problems, buffers on disk and keeps the network out of the request path. A direct HTTP handler is reasonable for a small number of high-value records, such as notifications to a chat channel.

Is logging.handlers.HTTPHandler suitable for production?

Not on its own. It sends one synchronous request per record from the calling thread, with no batching and no timeout by default, so a slow endpoint stalls every thread that logs. Behind a queue listener, with a batching subclass and a timeout, it becomes usable.

What happens when the webhook endpoint is down?

With a bounded queue in front, records accumulate until the queue is full and are then dropped, and the application is unaffected. Without one, every logging call waits for the connection to time out, which can take down the service that was trying to report a problem.

How do I avoid flooding a chat channel during an incident?

Rate limit and aggregate. Send at most a few messages per minute, and when records are suppressed, send a summary of how many. A channel receiving hundreds of identical alerts during an outage is both useless and likely to throttle the sender.

Related Guides