Python Logging Mastery

The Story That Explains Why Logging Exists

Python's logging module is the standard, thread-safe, level-aware, configurable, production-ready way to record what your program is doing. It ships with the standard library — no pip install required — and every serious Python codebase uses it.

Why print() Fails in Production

Every Python developer starts with print(). Every senior Python developer eventually rips them all out. Here's why.

The Five Log Levels

Every log message carries a severity. Python defines five standard levels — each with a numeric value. When the logger's level is set, anything below that threshold is silently dropped.

Quick Demo of All Five Levels

import logging

# The simplest possible setup
logging.basicConfig(level=logging.DEBUG)

logging.debug("Loop iteration i=5, value=42")
logging.info("Server started on port 8000")
logging.warning("Disk usage at 82%")
logging.error("Failed to send email to user@example.com")
logging.critical("Database connection pool exhausted!")

The Anatomy of a Logging System

Python's logging module is built from four cooperating parts. Once you see the wiring, everything else becomes obvious.

Visual Flow — From log.info() to Disk

Getting a Real Logger — Never Use the Root

Beginners call logging.info(...) directly, which uses the root logger. Professionals never do this. Instead, every module creates its own named logger with logging.getLogger(__name__). This gives you free hierarchy, free filtering, and free source identification.

# myapp/db.py
import logging

log = logging.getLogger(__name__)   # -> 'myapp.db'

def query(sql):
    log.debug("Executing SQL: %s", sql)
    # ... run query ...
    log.info("Query complete in %d ms", 42)

The Logger Hierarchy

Formatters — Making Logs Readable

A raw INFO:root:User logged in line is not enough for production. You want timestamp, level, source, thread, and message. Formatters use placeholders from the LogRecord attributes.

import logging

FMT = "%(asctime)s | %(levelname)-8s | %(name)s:%(lineno)d | %(message)s"
DATE = "%Y-%m-%d %H:%M:%S"

logging.basicConfig(level=logging.INFO, format=FMT, datefmt=DATE)

log = logging.getLogger("shop.checkout")
log.info("Order %s placed for $%.2f", "ORD-8821", 129.95)
log.warning("Coupon SUMMER25 expired but was accepted")

Handlers — Sending Logs Everywhere

A handler is a destination. The most common ones ship built-in with Python.

Full Setup — Console + Rotating File

import logging
from logging.handlers import RotatingFileHandler

# 1. Get a named logger
log = logging.getLogger("myapp")
log.setLevel(logging.DEBUG)   # capture everything at the logger

# 2. Build a formatter
fmt = logging.Formatter(
    "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)

# 3. Console handler -> only WARNING and up on screen
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(fmt)

# 4. File handler -> everything DEBUG+ to disk, rotate at 5MB, keep 5 backups
file_h = RotatingFileHandler(
    "myapp.log", maxBytes=5_000_000, backupCount=5
)
file_h.setLevel(logging.DEBUG)
file_h.setFormatter(fmt)

# 5. Attach both handlers
log.addHandler(console)
log.addHandler(file_h)

# Demo
log.debug("This goes to FILE only (console is WARNING+)")
log.info("This goes to FILE only")
log.warning("This goes to BOTH file and console")
log.error("This goes to BOTH file and console")

Logging Exceptions — Get the Full Traceback

A bare log.error("Something failed") loses the exception. Use log.exception() or exc_info=True to capture the full traceback.

import logging
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)

def divide(a, b):
    try:
        return a / b
    except ZeroDivisionError:
        # Automatically appends the traceback
        log.exception("Failed to divide %s by %s", a, b)
        return None

divide(10, 0)

dictConfig — The Production Pattern

For real apps you don't wire handlers with Python code — you describe the entire logging setup as a dictionary and pass it to logging.config.dictConfig(). This lives in one file and can be loaded from YAML/JSON at startup.

import logging.config

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,

    "formatters": {
        "standard": {
            "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        },
        "detailed": {
            "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(filename)s:%(lineno)d | %(message)s",
        },
    },

    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "formatter": "standard",
        },
        "file": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "DEBUG",
            "formatter": "detailed",
            "filename": "app.log",
            "maxBytes": 5_000_000,
            "backupCount": 5,
        },
        "error_file": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "ERROR",
            "formatter": "detailed",
            "filename": "errors.log",
            "maxBytes": 5_000_000,
            "backupCount": 10,
        },
    },

    "loggers": {
        "myapp": {
            "handlers": ["console", "file", "error_file"],
            "level": "DEBUG",
            "propagate": False,
        },
        "urllib3": {                # silence chatty 3rd-party libs
            "level": "WARNING",
        },
    },

    "root": {
        "handlers": ["console"],
        "level": "WARNING",
    },
}

logging.config.dictConfig(LOGGING)

log = logging.getLogger("myapp.orders")
log.info("Order #100 processed")
log.error("Payment gateway timeout for order #101")

Real-World Example — Flask Web App Logging

import logging
import logging.config
import time
from flask import Flask, request, g

# --- Logging setup ---
logging.config.dictConfig({
    "version": 1,
    "formatters": {"default": {
        "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s"
    }},
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "default",
        },
        "file": {
            "class": "logging.handlers.TimedRotatingFileHandler",
            "filename": "access.log",
            "when": "midnight",
            "backupCount": 14,
            "formatter": "default",
        },
    },
    "root": {"handlers": ["console", "file"], "level": "INFO"},
})

app = Flask(__name__)
log = logging.getLogger("shop.api")

@app.before_request
def _start_timer():
    g.start = time.perf_counter()

@app.after_request
def _log_request(response):
    elapsed_ms = (time.perf_counter() - g.start) * 1000
    log.info(
        "%s %s -> %d in %.1fms (ip=%s)",
        request.method, request.path, response.status_code,
        elapsed_ms, request.remote_addr,
    )
    return response

@app.route("/order/<int:order_id>")
def get_order(order_id):
    log.debug("Fetching order %s from DB", order_id)
    if order_id < 0:
        log.warning("Rejecting negative order id: %s", order_id)
        return {"error": "bad id"}, 400
    return {"id": order_id, "total": 42.00}

@app.errorhandler(Exception)
def _unhandled(err):
    log.exception("Unhandled error on %s", request.path)
    return {"error": "internal"}, 500

if __name__ == "__main__":
    app.run(port=8000)

Structured (JSON) Logging

Modern log aggregators (Datadog, Splunk, ELK, Loki) prefer JSON — one object per line, keys as searchable fields. Ship it with the python-json-logger package.

# pip install python-json-logger
import logging
from pythonjsonlogger import jsonlogger

log = logging.getLogger("shop")
log.setLevel(logging.INFO)

handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter(
    "%(asctime)s %(levelname)s %(name)s %(message)s"
))
log.addHandler(handler)

# Attach extra structured fields via `extra=`
log.info("Order placed", extra={
    "order_id": "ORD-8821",
    "user_id": 42,
    "amount": 129.95,
    "currency": "USD",
})

Performance — The Two Golden Rules

Common Pitfalls & How to Avoid Them

logging vs print vs loguru — Quick Comparison

Golden Rules