Cross-Validation
Press Next → or use ← → arrow keys
What is Cross-Validation, And Why Do We Bother?
Cross-Validation rotates the test set through the data multiple times, then averages the results — giving you a stable estimate with a real error bar instead of one lucky (or unlucky) number.
Overfitting detection — exposes the gap between "great on training" and "grim on new data". Lucky splits — averages away the randomness of any single partition. Data utilisation — every sample gets to be tested exactly once, every sample trains k−1 times.
K-Fold — The Rotating Test Set
Shuffle N samples, split into k folds. On each of k iterations, one fold is held out as test — the other k−1 train the model. Report mean ± std across all k scores. A high std is a red flag: your model behaves inconsistently across data slices.
The Family — Six CV Methods You Should Know
Classification → Stratified. Grouped samples → GroupKFold. Time-ordered → TimeSeriesSplit. Everything else → k=5 standard. Anything else is a special case.
Stratified K-Fold — Why It's The Default for Classification
Stratified K-Fold prevents random bad splits that would leave a fold with too few minority-class
samples to learn from. Every fold gets exactly the same class ratio as the full dataset.
StratifiedKFold is the default in sklearn's cross_val_score for classifiers — for good reason.
TimeSeries Split — Never Train On The Future
Standard K-Fold on time-ordered data lets the model train on future observations and predict the
past — the exact opposite of production. CV will look brilliant; deployment will collapse.
Use TimeSeriesSplit for anything sequential: stock prices, sales, weather, telemetry, logs.
Group K-Fold — When Samples Come In Clusters
Same patient in train + test → optimistic scores by 10–30 points. Same customer's transactions.
Same user's clicks. Same video's frames. If your samples cluster naturally, use GroupKFold.
Identify the grouping structure before you pick a CV method.
Nested CV — Honest Numbers When You Tune Hyperparameters
If you pick hyperparameters by CV score, then report that same CV score, you've overfit to the folds themselves. Nested CV separates the two: inner loop picks params, outer loop evaluates — never contaminated by the search.
Choosing K — The Bias–Variance–Compute Triangle
| k | Train per fold | Bias | Variance | Compute | Best For |
|---|---|---|---|---|---|
| k = 3 | 67% | High | Low | Cheap | Very large datasets, prototyping |
| k = 5 | 80% | Moderate | Low | 3× cheap | Default · 100–10,000 samples |
| k = 10 | 90% | Low | Medium | 2× k=5 | Research · tight comparisons |
| k = N (LOOCV) | ~100% | Minimal | High | N fits | Datasets under 100 samples |
Start with k = 5. Move to k = 10 only when you need tighter CIs and can afford double the compute. Reach for LOOCV only when the dataset is genuinely tiny (< 100 samples). Never use k = 2.
Preprocessing Leakage — The Silent Score Inflator
# ❌ WRONG — leaks test-fold statistics into training X_scaled = StandardScaler().fit_transform(X) # fits on ALL data! scores = cross_val_score(model, X_scaled, y, cv=5) # ✅ RIGHT — scaler refits on each training fold only pipe = Pipeline([('scaler', StandardScaler()), ('model', SVC())]) scores = cross_val_score(pipe, X, y, cv=5)
Scaling, imputation, feature selection, target encoding — anything that learns from data — must sit inside a Pipeline. That way it only sees the current training fold and transforms the test fold blindly. Outside the pipeline, you leak.
Implementation — The sklearn CV Toolkit
# ── Single metric ───────────────────────────────── from sklearn.model_selection import cross_val_score, StratifiedKFold from sklearn.pipeline import Pipeline skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(pipe, X, y, cv=skf, scoring='roc_auc') print(f"AUC = {scores.mean():.3f} ± {scores.std():.3f}") # ── Multi-metric with train scores & timing ────── from sklearn.model_selection import cross_validate results = cross_validate( pipe, X, y, cv=skf, scoring=['accuracy', 'f1', 'roc_auc'], return_train_score=True, n_jobs=-1, ) # ── Nested CV — honest tuning + evaluation ─────── from sklearn.model_selection import GridSearchCV inner = StratifiedKFold(n_splits=3, shuffle=True, random_state=1) outer = StratifiedKFold(n_splits=5, shuffle=True, random_state=2) grid = GridSearchCV(pipe, param_grid, cv=inner, scoring='roc_auc') nested = cross_val_score(grid, X, y, cv=outer, scoring='roc_auc') print(f"Honest AUC = {nested.mean():.3f} ± {nested.std():.3f}")
Set n_jobs=-1 to parallelise across every CPU core.
Set random_state on every splitter so folds are reproducible — mean ± std is worth reporting
only if others can recreate exactly your split.
The Decision Tree — Which CV Should You Use?
TimeSeriesSplit. Never shuffle.
Stock prices, sales, telemetry, logs — anything with a timestamp.
GroupKFold. Multiple scans per patient,
transactions per customer, frames per video. Keep the group together.
StratifiedKFold. Preserves class ratios; zero downside.
KFold(n_splits=5, shuffle=True).
Nested CV.
Otherwise your reported score is inflated by 1–5%.
RepeatedStratifiedKFold(5×10)
for 50 scores and tight confidence intervals.
Common Pitfalls — What Silently Breaks Your Numbers
Pipeline.TimeSeriesSplit. Non-negotiable.Golden Rules — 1 to 4
Golden Rules — 5 to 8
GroupKFold. Time-ordered? TimeSeriesSplit.
Get this wrong and every metric downstream lies.
CV — Turning A Lucky Number Into An Honest One
You now understand how K-Fold rotates, why stratification matters, how time-series and grouped data demand different splitters, and why Nested CV is the gold standard when tuning. Every honest ML performance number you produce from now on rides on these ideas.
Study hyperparameter tuning (GridSearch, RandomSearch, Optuna), then Learning Curves to diagnose bias vs variance, then bootstrapping for tighter CIs. Practice by re-running any past Kaggle notebook with proper Pipeline + Nested CV — the honest scores will humble you.
🔄 End of tutorial · Press ← to review, or click Restart