Ridge · Lasso · ElasticNet
Press Next → or use ← → arrow keys
Why Regularisation? — When Plain OLS Breaks
Then you hit the test set: R² = 0.71. Classic overfitting. Two feature columns are strongly correlated? OLS's coefficients swing wildly with any change in data — the multicollinearity crisis. Regularisation is the fix: add a penalty for large coefficients, and the model is forced to prefer modest weights that generalise.
Overfitting — coefficients grow enormous to hug the training data. Multicollinearity — correlated features produce unstable, high-variance coefficients. Both dissolve when you add a penalty term to the cost function that discourages large weights.
The Two Penalties — Squared vs Absolute
The regulariser tells the model: "You may use any coefficients you want — but the bigger they are, the higher the tax you pay." Only truly useful features can afford to keep large weights.
Coefficient Shrinkage — Three Regressions, Same Data
OLS lets coefficients grow wild. Ridge shrinks them all evenly, keeping every feature. Lasso zeros out the ones the data doesn't need — a built-in feature selector. Same data, three very different models.
Why Lasso Zeros — The L1 Diamond vs L2 Circle
Regularisation constrains coefficients to a shape. The RSS-minimising point wants to sit as close to OLS as possible — so it touches the constraint region at whichever point is closest. A smooth circle touches at a spot where both coefficients are non-zero. A diamond with sharp axis corners almost always touches on a corner — where one coefficient is exactly zero. That's why Lasso does feature selection and Ridge doesn't.
Coefficient Paths — As α Grows, Weights Shrink
Ridge paths smoothly bend toward zero — no coefficient ever hits it. Lasso paths stair-step: at each α threshold, another coefficient snaps to zero and stays there. The Lasso plot is the feature-selection story: read off which α gives you the number of features you want.
Tuning α — The Sweet Spot Between Bias & Variance
| α value | Effect on model | Symptom |
|---|---|---|
| α = 0 | No regularisation · identical to OLS | High variance · overfits |
| α small (10⁻³–10⁻¹) | Gentle shrinkage | Balanced · usually the sweet spot |
| α moderate (1–10) | Meaningful pull on coefficients | Robust to noisy data |
| α very large (100+) | All weights crushed toward 0 | High bias · underfits |
The optimal α depends entirely on your data. Sweep a log-spaced grid (e.g. np.logspace(-3, 4, 100))
via RidgeCV or LassoCV. The valley of the test-error curve is your target.
Lasso In Action — Automatic Feature Selection
Lasso regularises AND selects features in one shot. No separate feature-selection step required. Redundant neighbourhood dummies, useless interactions, ID-like columns — the L1 penalty snips them all to zero. The surviving 34 features form a simpler, faster, more explainable model.
The Three Constraint Shapes — Ridge · Lasso · ElasticNet
| Model | Penalty | Output | Best For |
|---|---|---|---|
| Ridge | L2 (squared) | Dense · all features kept | Correlated features · when everything matters a little |
| Lasso | L1 (absolute) | Sparse · auto-selection | High-dim data · few real predictors · interpretability |
| ElasticNet | L1 + L2 mix | Sparse but grouped | Correlated features AND need selection · genomics · text |
Scaling — The Non-Negotiable First Step
Fix: standardise every feature to mean 0, std 1 before fitting Ridge/Lasso.
StandardScaler is one line. Skip it and every coefficient is meaningless.
scaler.fit_transform(X_train), then scaler.transform(X_test) —
never re-fit on the full data. Fitting on the combined dataset leaks test statistics into training
and inflates every score. Better still: wrap it in a Pipeline and let cross-validation
handle it automatically.
Implementation — RidgeCV, LassoCV, ElasticNetCV
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline import numpy as np alphas = np.logspace(-3, 4, 100) # log grid — mandatory # ── Ridge — auto-tuned via CV ──────────────────── ridge = Pipeline([ ('scaler', StandardScaler()), ('ridge', RidgeCV(alphas=alphas, cv=10)), ]).fit(X_train, y_train) print(f"best α = {ridge.named_steps['ridge'].alpha_}") # ── Lasso — needs max_iter bumped up ──────────── lasso = Pipeline([ ('scaler', StandardScaler()), ('lasso', LassoCV(cv=10, max_iter=10000, random_state=42)), ]).fit(X_train, y_train) sparse = np.sum(lasso.named_steps['lasso'].coef_ != 0) print(f"features kept: {sparse}") # ── ElasticNet — best of both worlds ──────────── enet = Pipeline([ ('scaler', StandardScaler()), ('enet', ElasticNetCV( l1_ratio=[.1, .3, .5, .7, .9], # search the mix too alphas=np.logspace(-3, 2, 60), cv=10, max_iter=10000, n_jobs=-1, )), ]).fit(X_train, y_train)
Wrapping the scaler + model in a Pipeline means the scaler refits on each CV fold's
training partition only — no leakage. Do it manually and your CV scores lie by 2–5%.
Common Pitfalls — What Silently Breaks Regularised Models
StandardScaler, the penalty punishes small-unit features unfairly. Every Ridge/Lasso run needs it.RidgeCV/LassoCV. The right α depends on your data — never guess.max_iter to 10,000+.Choosing — Which Regularisation For Which Problem?
Let cross-validation search across l1_ratio values — if L1 wins, you get sparsity;
if L2 wins, you get stability. Either way, the CV picks the right blend for your data.
Golden Rules — 1 to 3
StandardScaler. Fit on training data only. Wrap in a Pipeline so CV
folds handle it correctly. Without scaling, the penalty punishes based on units, not signal.
RidgeCV, LassoCV, or
ElasticNetCV. Manual α picking is guesswork.
Golden Rules — 4 to 6
max_iter for Lasso.
Coordinate descent is iterative. If you see a ConvergenceWarning, raise to 10,000+
or scale features better. Silent non-convergence gives silently wrong coefficients.
✅ Features standardised in a Pipeline · ✅ α tuned by CV on log grid · ✅ Ridge vs Lasso vs ElasticNet chosen by problem structure · ✅ Coefficient path inspected · ✅ Convergence confirmed · ✅ Intercept unpenalised.
Ridge · Lasso — Two Penalties That Save Linear Models
You now understand why plain OLS breaks on wide data, how the L2 circle differs from the L1 diamond, why Lasso produces sparse models and Ridge doesn't, and how α steers the bias-variance dial. Every regularised model you build — LogisticRegression with penalties, GLMs, neural-network weight decay — rests on exactly these ideas.
Study Logistic Regression with L1/L2 for classification, then Bayesian Ridge for uncertainty quantification, then Group Lasso for feature-group selection. Practise on Kaggle's House Prices — the classic testbed for regularised regression.
⚖️ End of tutorial · Press ← to review, or click Restart