Gradient Boosting

The Story That Explains Gradient Boosting

Gradient Boosting is a sequential ensemble method that builds a series of weak decision trees — typically shallow — where each tree is trained to correct the cumulative prediction error of all previous trees. Unlike Random Forest (which builds trees independently and in parallel), Gradient Boosting is inherently sequential — each tree depends on what came before it.

Boosting vs Bagging — Two Different Philosophies

Both boosting and bagging are ensemble methods — they combine many weak models. But how they combine them is fundamentally different.

The Algorithm — Step by Step

Let's walk through the full algorithm using a regression problem — predicting house prices. Each step is precise but explained in plain English.

The Mathematics — What "Gradient" Actually Means

Gradient Boosting is rooted in functional gradient descent. Instead of updating model parameters with gradients (like in neural networks), it updates the prediction function itself in the direction that reduces loss the most.

Visual Diagram — Sequential Boosting Process

Loss Functions — The Heart of Flexibility

One of Gradient Boosting's greatest strengths is that it works with any differentiable loss function. The loss function determines what kind of "mistake" the model measures and minimises.

Key Hyperparameters — The Control Panel

🍎 Family 1 — Ensemble Structure

🌵 Family 2 — Tree Structure (Complexity)

🎗 Family 3 — Stochasticity (Randomness)

Shrinkage — The Single Most Important Tuning Decision

Stochastic Gradient Boosting — Adding Randomness

Instead of using the full training set for each tree, use a random subsample of rows (without replacement). This is controlled by the subsample parameter. Setting it below 1.0 creates Stochastic Gradient Boosting.

Overfitting and Regularisation in Gradient Boosting

Python — sklearn GradientBoostingClassifier

🛠️ Full Classification Pipeline: Credit Card Fraud Detection

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=8000, n_features=20,
    n_informative=12, n_redundant=4,
    weights=[0.95, 0.05],   # 95% legit, 5% fraud
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

gb = GradientBoostingClassifier(
    n_estimators=500,          # many trees — early stopping will prune
    learning_rate=0.05,        # small steps, better generalisation
    max_depth=4,               # shallow trees prevent overfitting
    min_samples_leaf=10,       # regularise leaf size
    subsample=0.75,            # stochastic GB: 75% row sample per tree
    max_features=0.5,          # 50% of features per split
    loss='log_loss',           # binary classification
    validation_fraction=0.1,   # 10% for early stopping
    n_iter_no_change=20,       # stop if no improvement for 20 rounds
    tol=1e-4,
    random_state=42
)
gb.fit(X_train, y_train)
print(f"Trees used (early stopping): {gb.n_estimators_}")
print(f"Test AUC-ROC: {roc_auc_score(y_test, gb.predict_proba(X_test)[:,1]):.4f}")
print(classification_report(y_test, gb.predict(X_test), digits=4))

📈 Regression Example: Predicting Energy Consumption

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error, r2_score

housing = fetch_california_housing()
X, y = housing.data, housing.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
gb_reg = GradientBoostingRegressor(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    max_features='sqrt',
    loss='huber',              # robust to outlier prices
    alpha=0.9,
    validation_fraction=0.1,
    n_iter_no_change=25,
    random_state=42
)
gb_reg.fit(X_train, y_train)
y_pred = gb_reg.predict(X_test)
print(f"R² Score : {r2_score(y_test, y_pred):.4f}")
print(f"MAE      : ${mean_absolute_error(y_test, y_pred) * 100_000:.0f}")
print(f"Trees    : {gb_reg.n_estimators_} (early stopped)")

LightGBM — Faster, Scalable Gradient Boosting

LightGBM is Microsoft's production-grade implementation. It is the same sequential residual-fitting logic, but with two critical engineering innovations that make it dramatically faster on large datasets.

import lightgbm as lgb
from sklearn.metrics import roc_auc_score, classification_report

train_data = lgb.Dataset(X_train, label=y_train)
val_data   = lgb.Dataset(X_test,  label=y_test, reference=train_data)

params = {
    'objective':         'binary',
    'metric':            'auc',
    'learning_rate':     0.05,
    'num_leaves':        31,         # primary complexity control for LGB
    'max_depth':         -1,         # -1 = no limit; num_leaves controls it
    'min_child_samples': 20,
    'feature_fraction':  0.7,
    'bagging_fraction':  0.8,
    'bagging_freq':      5,
    'lambda_l1':         0.1,
    'lambda_l2':         1.0,
    'verbosity':         -1,
    'random_state':      42
}
callbacks = [lgb.early_stopping(stopping_rounds=50), lgb.log_evaluation(period=100)]
model = lgb.train(params, train_data, num_boost_round=2000,
                   valid_sets=[val_data], callbacks=callbacks)

y_prob = model.predict(X_test)
print(f"Best iteration : {model.best_iteration}")
print(f"AUC-ROC        : {roc_auc_score(y_test, y_prob):.4f}")

Feature Importance in Gradient Boosting

Gradient Boosting vs Random Forest — Full Comparison

Golden Rules