Ensemble Learning
Press Next → or use ← → arrow keys
Why Combine Models At All?
Ensemble learning is that fair, formalised. Train many models that each make different mistakes, then combine them. As long as their errors are uncorrelated, the ensemble is far more accurate — and more stable — than any single member.
A weak learner only needs to be slightly better than a coin flip. Combine enough of them, with uncorrelated errors, and the ensemble converges toward near-perfect accuracy. Diversity is the whole game.
The Maths — Uncorrelated Errors Cancel
Ensemble variance = ρσ² + (1−ρ)/N · σ², where ρ is the correlation between models, σ² their individual variance, and N the count. As N grows, the second term vanishes — but the first term, ρσ², only shrinks if models are uncorrelated. That's the mathematical proof that diversity beats individual accuracy.
The Three Families At A Glance
Bagging trains many models in parallel and averages them — killing variance. Boosting trains models in sequence, each fixing the last one's mistakes — killing bias. Stacking trains a meta-model to blend diverse base learners — attacking both.
Bagging — Parallel Independence
Each tree trains on a different bootstrap sample (rows drawn with replacement — ~63% unique).
Random Forest adds a random feature subset at every split for extra diversity. Because the trees are
independent, training is trivially parallel (n_jobs=-1), and averaging
their uncorrelated errors crushes variance. The ~37% left-out rows give free OOB validation.
Boosting — Sequential Error Correction
Each learner up-weights (AdaBoost) or fits the residuals (Gradient Boosting) of the previous one, so
the ensemble relentlessly reduces bias. But it cannot be parallelised, and it
will overfit given enough rounds. Always keep a validation set and use
early_stopping_rounds. The key knob is learning_rate × n_estimators.
The Boosting Family — AdaBoost to CatBoost
| Method | Core Mechanism | Best For |
|---|---|---|
| AdaBoost | Re-weights misclassified samples each round | Clean, balanced binary classification |
| Gradient Boosting | Fits the residual errors of previous trees | Flexible; subsampling tames noise |
| XGBoost | GBM + histogram splits, sparsity handling, regularisation | Production & competitions |
| LightGBM | Leaf-wise growth, histogram binning | Very large datasets · speed |
| CatBoost | Ordered boosting, native categorical handling | Categorical-heavy data · little tuning |
Gradient-boosted trees — especially XGBoost and LightGBM — win the
majority of tabular ML competitions. When you need the last 2–3% of accuracy that Random Forest can't
reach, this is where you go. Pair with RandomizedSearchCV and early stopping.
Stacking — A Meta-Learner On Top
The meta-learner must train on predictions the base models made on data they didn't see. Use K-fold: each base learner trains on K−1 folds and predicts the held-out fold — these out-of-fold predictions become the meta-features. Skip this and the meta-learner sees artificially perfect predictions, overfits, and collapses in production.
Voting — The Quick Ensemble
A model that says "90% spam" should count for more than one that barely says "51% spam". Soft voting
respects that confidence; hard voting flattens both to a single tick. Reach for soft voting whenever
your base models expose predict_proba.
Which Error Does Each Family Fix?
| Problem | Cause | Ensemble Fix |
|---|---|---|
| High variance | Deep model memorises noise | Bagging / Random Forest — average uncorrelated trees |
| High bias | Shallow model underfits | Boosting — sequentially correct residuals |
| Both | No single architecture is best | Stacking — learn the optimal blend |
Which Ensemble When?
Start with Random Forest to establish a solid baseline. If you need more, move to gradient boosting. Only if the last couple of percent genuinely matters do you reach for stacking — it's the most work for the smallest gain.
Implementation — All Four In Sklearn
from sklearn.ensemble import ( RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, VotingClassifier, StackingClassifier) from sklearn.linear_model import LogisticRegression from sklearn.svm import SVC from sklearn.neighbors import KNeighborsClassifier # ── BAGGING · the default baseline ─────────────── rf = RandomForestClassifier(n_estimators=300, n_jobs=-1, oob_score=True) # ── BOOSTING · higher ceiling · needs early stopping ── import xgboost as xgb gbm = xgb.XGBClassifier( n_estimators=1000, learning_rate=0.05, max_depth=4, subsample=0.8, early_stopping_rounds=30, eval_metric='auc') gbm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)]) # ── VOTING · soft = average probabilities ──────── vote = VotingClassifier(estimators=[ ('rf', rf), ('svc', SVC(probability=True)), ('knn', KNeighborsClassifier()) ], voting='soft', n_jobs=-1) # ── STACKING · out-of-fold preds via cv, simple meta ── stack = StackingClassifier( estimators=[('rf', rf), ('svc', SVC(probability=True)), ('knn', KNeighborsClassifier())], final_estimator=LogisticRegression(), cv=5, n_jobs=-1) # cv=5 handles OOF automatically
Always pass n_jobs=-1 to parallelise (bagging, voting, stacking). Always give boosting an
eval_set and early_stopping_rounds — it will overfit otherwise.
sklearn's StackingClassifier(cv=5) generates the out-of-fold predictions for you, so leakage
is handled correctly.
Common Pitfalls — What Silently Breaks Ensembles
early_stopping_rounds=30.permutation_importance or SHAP for attribution.Golden Rules — 1 to 4
early_stopping_rounds — non-negotiable for GBM / XGBoost / LightGBM.
cv=5 does this) or the meta-learner
overfits to leaked, artificially-perfect inputs.
Golden Rules — 5 to 7
n_jobs=-1.
Bagging, voting and stacking parallelise trivially across cores. There's no reason to train on a
single core when the whole machine is available.
✅ RF baseline established · ✅ base learners genuinely diverse · ✅ boosting has early stopping · ✅ stacking uses out-of-fold preds · ✅ n_jobs=-1 set · ✅ metric matches the problem, not just accuracy.
Ensembles — Many Models, One Better Answer
You now understand why combining models beats any single one, how bagging kills variance, how boosting kills bias, how stacking learns the best blend, and why diversity is the whole game. These techniques power the majority of winning tabular-ML systems in production today.
Deep-dive Random Forest and XGBoost / LightGBM / CatBoost individually, then study SHAP for ensemble interpretability and Optuna for hyperparameter search. Practise on Kaggle's Titanic, House Prices and any tabular competition — ensembles dominate the leaderboards.
🎭 End of tutorial · Press ← to review, or click Restart