Ridge & Lasso Regression

The Story That Explains Everything

In machine learning, overfitting happens when a model learns the training data too well — including its noise, quirks, and random fluctuations. It performs brilliantly on data it has seen and catastrophically on data it has not. Regularisation adds a penalty term to the loss function that discourages large, complex coefficients, giving us models that generalise far better to new data.

The Problem — Why Plain Linear Regression Fails

Standard Ordinary Least Squares (OLS) regression minimises only the residual sum of squares. It will assign whatever coefficients minimise training error, even if those coefficients are enormous and unstable.

Ridge Regression — L2 Regularisation

Ridge Regression modifies the cost function by adding the sum of squared coefficients multiplied by a penalty strength alpha (α):

Lasso Regression — L1 Regularisation

Lasso (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute values of coefficients as its penalty:

The Geometry — Why L1 Zeros Coefficients and L2 Doesn't

The deepest insight into Ridge vs Lasso comes from visualising their constraint regions in coefficient space. This is the most important diagram in all of regularisation theory.

Interactive 3D Penalty Surfaces

The best way to build intuition is to see what L1 and L2 penalties actually look like as surfaces in 3D. Rotate and explore the shapes below.

Ridge vs Lasso — Side-by-Side Comparison

The Alpha Hyperparameter — Bias-Variance Tradeoff

Alpha (α) is the single most important hyperparameter in regularised regression. It controls the entire bias-variance tradeoff.

Why Feature Scaling is Mandatory

Python — Ridge Regression with Cross-Validation

Let's build a complete Ridge regression example using the California Housing dataset, with proper scaling, cross-validation alpha selection, and coefficient analysis.

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_squared_error, r2_score

# ── 1. Load data ───────────────────────────────────────
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
feature_names = data.feature_names

# ── 2. Train / test split ───────────────────────────────
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# ── 3. CRITICAL: Scale features ─────────────────────────
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)  # fit ONLY on train
X_test_sc  = scaler.transform(X_test)       # transform test with train stats

# ── 4. RidgeCV auto-selects best alpha ──────────────────
alphas = np.logspace(-3, 4, 100)  # 0.001 to 10,000
ridge_cv = RidgeCV(alphas=alphas, cv=10, scoring='neg_mean_squared_error')
ridge_cv.fit(X_train_sc, y_train)
print(f"Best alpha: {ridge_cv.alpha_:.4f}")

# ── 5. Evaluate ──────────────────────────────────────────
y_pred = ridge_cv.predict(X_test_sc)
mse    = mean_squared_error(y_test, y_pred)
r2     = r2_score(y_test, y_pred)
print(f"Test RMSE: {mse**0.5:.4f}")
print(f"Test R²  : {r2:.4f}")

# ── 6. Coefficient analysis ──────────────────────────────
coef_df = pd.DataFrame({
    'Feature'   : feature_names,
    'Coefficient': ridge_cv.coef_,
    'Abs_Coef'  : np.abs(ridge_cv.coef_)
}).sort_values('Abs_Coef', ascending=False)

print("\nRidge Coefficients (scaled features):")
print(coef_df.to_string(index=False))

Python — Lasso Regression with Feature Selection

Now the same dataset with Lasso — watch how it performs automatic feature selection by zeroing out unimportant coefficients.

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Lasso, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# ── Load, split, scale (same as Ridge) ──────────────────
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
feature_names = data.feature_names

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# ── LassoCV: 10-fold CV over 100 alpha values ────────────
lasso_cv = LassoCV(
    alphas=None,        # auto-generates log-spaced alphas
    cv=10,
    max_iter=10000,     # Lasso needs more iterations
    random_state=42
)
lasso_cv.fit(X_train_sc, y_train)
print(f"Best alpha: {lasso_cv.alpha_:.6f}")

# ── Evaluate ─────────────────────────────────────────────
y_pred = lasso_cv.predict(X_test_sc)
mse    = mean_squared_error(y_test, y_pred)
r2     = r2_score(y_test, y_pred)
print(f"Test RMSE : {mse**0.5:.4f}")
print(f"Test R²   : {r2:.4f}")

# ── Feature selection report ─────────────────────────────
nonzero_mask = lasso_cv.coef_ != 0
print(f"\nFeatures selected: {nonzero_mask.sum()} / {len(feature_names)}")
print("\nLasso Coefficients:")
for name, coef in zip(feature_names, lasso_cv.coef_):
    status = 'SELECTED' if coef != 0 else 'ZEROED'
    print(f"  {name:12s}: {coef:8.4f}  [{status}]")

Coefficient Path — How Coefficients Change with Alpha

One of the most revealing visualisations is the regularisation path: plotting each coefficient's value as alpha sweeps from 0 (OLS) to large (all-zero). The difference between Ridge and Lasso paths is dramatic.

ElasticNet — The Best of Both Worlds

When you want Ridge's stability with correlated features and Lasso's feature selection, ElasticNet combines both penalties with two hyperparameters.

from sklearn.linear_model import ElasticNetCV
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
import numpy as np

data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_tr = scaler.fit_transform(X_train)
X_te = scaler.transform(X_test)

# ElasticNetCV searches both alpha AND l1_ratio
enet = ElasticNetCV(
    l1_ratio=[.1, .3, .5, .7, .9, .95, 1.0],  # test many mixing ratios
    alphas=np.logspace(-3, 2, 60),
    cv=10,
    max_iter=10000,
    random_state=42
)
enet.fit(X_tr, y_train)

print(f"Best alpha     : {enet.alpha_:.6f}")
print(f"Best l1_ratio  : {enet.l1_ratio_:.2f}")
print(f"Test R²        : {r2_score(y_test, enet.predict(X_te)):.4f}")
print(f"Features zeroed: {(enet.coef_==0).sum()}")

When to Use Ridge vs Lasso vs ElasticNet

Real-World Story — House Prices and 80 Features

# Lasso path for feature selection — Ames Housing workflow sketch
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
import numpy as np

# Assume X_train_raw, X_test_raw, y_train are loaded
num_features = ['LotArea', 'GrLivArea', 'OverallQual', 'YearBuilt']
cat_features = ['Neighborhood', 'BldgType', 'HouseStyle']

preprocessor = ColumnTransformer([
    ('num', StandardScaler(), num_features),
    ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), cat_features)
])

pipe = Pipeline([
    ('prep',   preprocessor),
    ('model',  LassoCV(cv=10, max_iter=20000, random_state=42))
])

pipe.fit(X_train_raw, np.log1p(y_train))  # log-transform skewed prices

lasso = pipe.named_steps['model']
n_total  = lasso.coef_.shape[0]
n_kept   = (lasso.coef_ != 0).sum()
print(f"Total features : {n_total}")
print(f"Features kept  : {n_kept} ({100*n_kept/n_total:.0f}%)")
print(f"Features zeroed: {n_total - n_kept} ({100*(n_total-n_kept)/n_total:.0f}%)")
print(f"Best alpha     : {lasso.alpha_:.6f}")

Common Mistakes — And How to Fix Them

Golden Rules