Bias · Variance · Overfitting
Press Next → or use ← → arrow keys
Four Words Every ML Practitioner Must Own
Ravi shoots five arrows and they land everywhere — one bullseye, one in the sky, three in different corners. Sometimes right, always unpredictable. That's high variance.
A good model is like the archer whose arrows cluster tightly around the bullseye — low bias, low variance. Everything that follows is how to become that archer.
The Dartboard — Four Ways A Model Can Fail
Cluster location reveals bias (are the darts near the bullseye?). Cluster spread reveals variance (how tightly do they group?). Every model you train lives somewhere on this board — your job is to move it toward the top-left.
Underfit vs Good Fit vs Overfit — In Pictures
| Model | Train Error | Test Error | Gap | Verdict |
|---|---|---|---|---|
| Straight line (deg 1) | 44 | 42 | small | High Bias · Underfit |
| Smooth curve (deg 3) | 10 | 13 | small | Sweet Spot ✅ |
| Wiggle (deg 15) | 0.2 | 75 | HUGE | High Variance · Overfit |
🎬 Feel The Bias — Cycle Through Complexities
Hold the data fixed. Click Next Model to increase complexity from a flat line all the way to a degree-15 wiggle. Watch the green curve pull closer to the true trend — that shrinking gap is bias decreasing in real time.
Constant Model degree 0 · flat line
The One Equation Ruling Every Model
If irreducible noise is 8%, your best possible test error is 8%. Any model achieving 2% train error on such data has memorised, not learned. Accept the floor.
The U-Curve — Complexity's Two-Sided Cost
Moving right along the complexity axis always trades bias for variance. The only ways out of the tradeoff: more data (curve flattens), regularisation (curve shifts), ensembles (average many models). Everything else is picking a point.
🎬 Feel The Variance — Cycle Through Data Samples
Two models — simple linear vs complex deg-15 — trained on the same underlying pattern. Click Next Sample to reshuffle the noise. The blue line barely twitches (low variance). The purple curve reinvents itself completely (high variance).
Simple Model linear · low complexity
Complex Model deg 15 · high complexity
Learning Curves — Your One-Chart Diagnostic
Both curves high, small gap → underfitting: add complexity or features. Both curves low, small gap → ship the model. Train low, val high, huge gap → overfitting: regularise, get more data, or simplify.
The Fix Map — Diagnose First, Then Cure
Throwing "more data" at high bias fails. Throwing "more complexity" at high variance backfires. Plot the learning curve first, then pick a remedy from the matching column.
Regularisation — Sliding Along The Curve
| λ value | Effect | Symptom |
|---|---|---|
| λ = 0 | No constraint · full complexity | High variance · overfitting risk |
| λ small (0.001–0.1) | Mild constraint · balanced | Sweet spot for most datasets |
| λ moderate (1–10) | Meaningful shrinkage | Good for noisy or wide feature sets |
| λ large (100+) | Weights crushed toward zero | High bias · underfitting risk |
Standard grid: [0.0001, 0.001, 0.01, 0.1, 1, 10, 100].
Use RidgeCV / LogisticRegressionCV / GridSearchCV — never eyeball λ.
Regularisation is the steering wheel of the bias-variance curve.
Ensembles — The Pragmatic Escape Route
A single model must pick one complexity level along the U. Ensembles combine many models with different biases/variances — the average lands lower than any single model can. This is why gradient-boosted trees still dominate tabular ML competitions.
Implementation — Plot The Curve, Tune The λ
# ── Learning curves — diagnose bias vs variance ── from sklearn.model_selection import learning_curve from sklearn.linear_model import Ridge sizes, train_err, val_err = learning_curve( Ridge(alpha=1.0), X, y, train_sizes=np.linspace(0.1, 1.0, 10), cv=5, scoring='neg_mean_squared_error', n_jobs=-1, ) # Plot mean of train_err and val_err vs sizes — read the pattern. # ── Validation curve — sweep complexity/regularisation ── from sklearn.model_selection import validation_curve train_err, val_err = validation_curve( Ridge(), X, y, param_name='alpha', param_range=np.logspace(-4, 3, 8), # log grid — always cv=5, scoring='neg_mean_squared_error', ) # ── Auto-pick λ with cross-validation ── from sklearn.linear_model import RidgeCV model = RidgeCV(alphas=np.logspace(-4, 3, 8), cv=5) model.fit(X_train, y_train) print(f"Best alpha: {model.alpha_}")
A learning curve (error vs training-set size) diagnoses bias vs variance. A validation curve (error vs a hyperparameter like λ or max_depth) picks the sweet spot. Both are one-liners in sklearn — no excuse to skip them.
Common Pitfalls — What Silently Breaks Models
Golden Rules — 1 to 3
Golden Rules — 4 to 6
RidgeCV / GridSearchCV. Never pick a single value from intuition.
✅ Learning curve plotted · ✅ Bias vs variance diagnosed · ✅ Remedy matches diagnosis ✅ λ chosen by CV on log grid · ✅ Ensemble considered · ✅ Std reported alongside mean.
Bias · Variance — The Landscape, Not A Problem
You now read the dartboard, the U-curve, and the learning-curve trio. You know which lever to pull for bias vs variance, and why ensembles cheat the tradeoff. Every model you tune from now on — linear, tree, or neural — lives inside this framework.
Study cross-validation (to measure variance rigorously), then Ridge, Lasso & ElasticNet in depth, then Random Forest & XGBoost. Practise on Kaggle's House Prices — the same dataset a linear, a tree and an ensemble will land in three different spots on the U-curve.
🎯 End of tutorial · Press ← to review, or click Restart