Random Forest
Press Next → or use ← → arrow keys
What Is A Random Forest?
A Random Forest assembles 300 jurors. Each one reads a different random slice of the evidence and considers only a random subset of questions. Their individual mistakes point in every direction, so when they vote, the errors cancel out and the truth emerges. That's the entire idea.
A Random Forest is an ensemble of decision trees, each trained on a bootstrap sample of the data with a random subset of features at every split. Classification predictions are the majority vote; regression predictions are the average across all trees.
The Architecture — From One Dataset To One Prediction
1️⃣ Start with the full dataset. 2️⃣ Draw many bootstrap samples (rows with replacement). 3️⃣ Grow one deep tree per sample, considering only a random subset of features at each split. 4️⃣ Aggregate the trees — vote for classification, average for regression. That's the whole model.
Bootstrap Sampling — Rows Drawn With Replacement
Every bootstrap sample has the same number of rows as the original dataset — but drawn with replacement. Some rows appear twice or three times; others don't appear at all. Those absent rows become each tree's out-of-bag test set.
Random Feature Subsets — The Second Diversity Trick
Bootstrap sampling shuffles the rows. Random feature subsampling shuffles the columns — at every single split, each tree considers only a random handful of features. This is what stops one dominant feature from bossing every tree the same way.
Plain bagging (row sampling only) gives you an ensemble of similar trees. Adding random feature selection at every split is what turns bagging into a Random Forest — and what makes it dramatically stronger than plain bagged trees.
Aggregation — Majority Vote & Averaging
| Task | Aggregation | Example |
|---|---|---|
| Classification | Majority vote | 4 of 5 trees say SPAM → SPAM |
| Classification (probabilities) | Average predicted probabilities | (0.9 + 0.85 + 0.3 + 0.7 + 0.95) / 5 = 0.74 → SPAM |
| Regression | Mean of predictions | (₹85L + ₹92L + ₹78L + ₹88L + ₹90L) / 5 = ₹86.6L |
Out-Of-Bag Score — Free Cross-Validation
Every tree naturally misses ~37% of the rows. Test each tree only on its missing rows,
aggregate the results — you get a validation score for free, using 100% of your data for training.
Enable it with oob_score=True. Read it via model.oob_score_. Ship the number.
Why It Works — Variance Reduction Made Visible
| Model | Bias | Variance | Test error trend |
|---|---|---|---|
| Shallow single tree | HIGH | low | Plateau — underfits |
| Deep single tree | low | HIGH | Overfits · very unstable |
| Random Forest (300 deep trees) | low-medium | LOW | Stable · production-ready ✅ |
Adding more trees to a Random Forest cannot increase test error — it only helps or plateaus. The variance-reduction ceiling flattens around 300–500 trees; past that you're paying for compute without gaining accuracy.
Hyperparameters — Seven Dials That Matter
| Parameter | Typical | Effect |
|---|---|---|
n_estimators | 300–500 | Number of trees. Bigger is safer; diminishing returns after 300. |
max_features | 'sqrt' · 'log2' | Features per split. Tune this first — biggest lever. |
max_depth | None | Depth cap. Leave unlimited unless memory-bound. |
min_samples_leaf | 1–10 | Min samples at any leaf. Increase to fight overfitting on noisy data. |
bootstrap | True | Row sampling with replacement. Disabling removes the core diversity mechanism. |
oob_score | True | Free unbiased validation. Set it always. |
class_weight | 'balanced_subsample' | Handles imbalanced classes automatically. |
1. max_features — biggest accuracy impact per unit of tuning effort.
2. min_samples_leaf — protects against overfitting.
3. n_estimators — increase last, until OOB score plateaus.
Everything else can safely stay at default.
Feature Importance — Which Predictors Actually Matter
Strengths & Weaknesses
n_jobs=-1 uses every core — 8-core machine trains 8× faster.max_depth or reduce n_estimators when RAM-bound.Random Forest — trees built independently in parallel. Forgiving of bad hyperparameters. Great baseline. Ship first, tune later. Gradient Boosting (XGBoost, LightGBM, CatBoost) — trees built sequentially, each correcting the previous one's mistakes. Higher peak accuracy but needs careful tuning and overfits if you're not watching. Use RF first; upgrade when you need the last 2–3%.
Implementation — RandomForestClassifier In A Dozen Lines
from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score, GridSearchCV from sklearn.inspection import permutation_importance # ── Sane production defaults ───────────────────── rf = RandomForestClassifier( n_estimators=300, max_features='sqrt', min_samples_leaf=2, oob_score=True, # free validation n_jobs=-1, # use every core class_weight='balanced_subsample', # imbalance handling random_state=42, ) rf.fit(X_train, y_train) print(f"OOB score: {rf.oob_score_:.4f}") # no test set needed print(f"CV score: {cross_val_score(rf, X, y, cv=5, n_jobs=-1).mean():.4f}") # ── Tune max_features first ───────────────────── grid = GridSearchCV(rf, param_grid={ 'max_features': ['sqrt', 'log2', 0.3, 0.5], 'min_samples_leaf': [1, 2, 5, 10], }, cv=5, scoring='roc_auc', n_jobs=-1).fit(X_train, y_train) # ── Trustworthy feature importance ────────────── perm = permutation_importance(rf, X_test, y_test, n_repeats=10, n_jobs=-1) for i in perm.importances_mean.argsort()[::-1]: print(f"{cols[i]:20s} {perm.importances_mean[i]:.4f}")
n_jobs=-1 — trains 8× faster on an 8-core box. There's no reason to leave it at 1.
oob_score=True — a free validation metric that costs nothing extra.
Skipping either is leaving performance on the table.
Common Pitfalls — What Silently Bites Practitioners
StandardScaler to RF adds noise and CPU cost with zero benefit.class_weight='balanced_subsample'.permutation_importance before dropping features.n_jobs at its default is wasting hours on multi-core hardware. Always n_jobs=-1.random_state, two runs give different scores and no one can reproduce your result.Golden Rules — 1 to 4
n_jobs=-1.
Trees train independently — parallelisation is essentially free. Not using every core is
throwing away 8× speed on modern hardware.
oob_score=True.
You get an unbiased validation metric for free using rows each tree never trained on.
No holdout split required.
StandardScaler for logistic
regression, SVM and KNN.
max_features, min_samples_leaf, n_estimators.
Feature subset size drives accuracy most. Leaf size fights overfitting. Add more trees last
until the OOB score stops improving.
Golden Rules — 5 to 7
class_weight='balanced_subsample'.
Report F1 and ROC-AUC — never accuracy. Consider probabilistic thresholds tuned on validation
for the operating point.
permutation_importance before acting on it.
MDI overrates numeric & high-cardinality columns. Permutation importance is slower but honest.
Drop features only when both agree.
✅ n_jobs=-1 · ✅ oob_score=True · ✅ random_state set · ✅ max_features tuned via CV · ✅ class_weight for imbalanced data · ✅ Permutation importance verified · ✅ OOB and test scores within 2%.
Random Forest — The Reliable Default Of Tabular ML
You now understand bagging, random feature subsampling, out-of-bag scoring, and why a forest crushes a single tree's variance. Random Forest is the reliable default for tabular ML — quick to train, forgiving of bad tuning, and often good enough to ship without a second thought.
Study Gradient Boosting (XGBoost, LightGBM, CatBoost) to squeeze out the final 2–3%. Then Extra Trees (a variance-happy cousin) and Isolation Forest (anomaly detection with the same architecture). Practise on Kaggle's Titanic, House Prices, and Bank Churn — RF is competitive on all three.
🌲 End of tutorial · Press ← to review, or click Restart