Decision Trees
Press Next → or use ← → arrow keys
What is a Decision Tree?
A Decision Tree is the same idea, formalised: a flowchart of yes/no questions the machine learns from your training data — no scaling, no engineering, no black box. Every prediction comes with a readable trace of the questions that led to it.
A Decision Tree is a supervised algorithm that models predictions as a hierarchy of if-else questions, learned automatically by scoring which split separates the classes best at every step.
Tree Anatomy — Root, Nodes, Leaves
Impurity — How A Split Is Scored
Both curves have the same shape — zero at the ends (pure), maximum in the middle (50/50 chaos). They almost always pick the same split in practice. Use Gini for speed; use Entropy when you want the extra sensitivity on skewed multi-class data.
Information Gain — The Split That Wins
For every candidate split, compute how much impurity drops. The one with the biggest drop becomes the split. On the classic Play-Tennis dataset [9 Yes, 5 No]:
| Feature | Weighted Child Entropy | Information Gain | Verdict |
|---|---|---|---|
| Outlook | 0.694 | 0.246 🏆 | Winner — becomes root |
| Humidity | 0.789 | 0.151 | Runner-up |
| Wind | 0.892 | 0.048 | Weak splitter |
| Temperature | 0.911 | 0.029 | Never selected |
Splitting on Outlook produces one perfectly pure child (Overcast: 4 Yes, 0 No → entropy 0). That single clean split alone recovers almost 25% of the parent entropy. Temperature's tiny 0.029 gain means it barely helps distinguish classes — the tree will ignore it.
Building The Tree — Play Tennis Result
All five leaves are pure (entropy = 0). Temperature was scored, deemed useless (IG = 0.029) and never included. The tree captured the pattern in just two splits — human-readable, testable, deployable.
The CART Algorithm — 4 Steps, Repeat
min_samples_split, max_depth reached, or no split improves impurity.CART picks the best split now, not the split that would enable the best future splits. A globally optimal tree is NP-hard. In practice, the greedy choice is good enough — especially inside ensembles that average many trees.
Decision Boundary — Axis-Aligned Rectangles
A tree can only split parallel to an axis (horizontal or vertical cuts in 2D). If your classes are truly separated by a diagonal, a single tree will need many stair-step splits to approximate it. Rotate features or switch to SVM / linear models when diagonal patterns dominate.
Depth vs Overfitting — The Tree's Achilles Heel
An unconstrained tree keeps splitting until every leaf is pure — memorising every training point, including the noise. That looks brilliant on train and collapses on test.
| max_depth | Train Accuracy | Test Accuracy | Gap | Verdict |
|---|---|---|---|---|
| 2 | 87% | 86% | 1% | Slight underfit |
| 4 | 93% | 90% | 3% | Sweet spot ✅ |
| 8 | 98% | 87% | 11% | Starting to overfit |
| None (unlimited) | 100% | 82% | 18% | Severe overfit |
Real data always has noise. A tree hitting 100% on training has memorised, not learned. Always constrain depth — start at 3, increase until validation accuracy plateaus.
Pruning — Trading Some Bias For A Lot Less Variance
Pre-pruning stops growth early using hyperparameters:
max_depth, min_samples_split, min_samples_leaf.
Post-pruning (Cost Complexity, ccp_alpha) grows the full tree first,
then collapses branches whose complexity penalty exceeds their gain. Both close the train-test gap.
Hyperparameters — The Six Dials That Matter
| Parameter | Default | Effect | Too Low | Too High |
|---|---|---|---|---|
max_depth | None | Max levels from root to leaf | Underfit | Overfit |
min_samples_split | 2 | Min samples to attempt a split | Splits noise | Stops early |
min_samples_leaf | 1 | Min samples in any final leaf | Single-sample leaves | Forces large leaves |
max_features | None | Features considered per split | Random | No benefit |
criterion | 'gini' | Split scoring function | — | — |
ccp_alpha | 0.0 | Post-pruning penalty strength | No pruning | Prunes to stump |
max_depth = 4, min_samples_split = 10, min_samples_leaf = 5,
criterion = 'gini'. Then let GridSearchCV refine around that on a
log grid. Tune on validation folds — never on the test set.
Feature Importance — What The Tree Actually Uses
Every split reduces impurity by some amount. Sum those reductions per feature → normalise → you get
each feature's share of the tree's decisions. In Ramesh's model,
credit score alone drives 58% of the predictions. Features with near-zero importance
— like num_accounts — can be dropped without hurting accuracy.
Strengths & Weaknesses
A single Decision Tree is rarely production-worthy. Its true power is as the atom of ensembles: Random Forest (bagging → crushes variance), Gradient Boosting / XGBoost / LightGBM (sequential correction → crushes bias). Both still win Kaggle tabular competitions in 2026.
Implementation — Scikit-learn In A Dozen Lines
from sklearn.tree import DecisionTreeClassifier, plot_tree from sklearn.model_selection import GridSearchCV, train_test_split # ── Fit with sane production defaults ──────────── dt = DecisionTreeClassifier( criterion='gini', max_depth=4, min_samples_split=10, min_samples_leaf=5, random_state=42, ) dt.fit(X_train, y_train) proba = dt.predict_proba(X_new) # calibrated probs # ── Inspect & visualise ───────────────────────── plot_tree(dt, feature_names=cols, class_names=['reject', 'approve'], filled=True) print(dict(zip(cols, dt.feature_importances_))) # ── Tune with GridSearchCV ────────────────────── grid = GridSearchCV( DecisionTreeClassifier(random_state=42), param_grid={ 'max_depth': [2, 3, 4, 5, 6], 'min_samples_split': [5, 10, 15, 20], 'min_samples_leaf': [2, 5, 10], 'criterion': ['gini', 'entropy'], }, cv=5, scoring='roc_auc', n_jobs=-1, ) grid.fit(X_train, y_train) print(grid.best_params_, grid.best_score_)
Always set random_state for reproducibility. Always cap max_depth in
production — the default (None) invites overfitting. Then let CV pick the exact value.
Common Pitfalls — What Bites Practitioners
max_depth=None lets trees grow until every leaf is pure → 100% train, disaster on test.class_weight='balanced' or resample. Otherwise the majority class dominates every split.Golden Rules — 1 to 3
max_depth.
Start at 3, increase by 1 until validation accuracy plateaus. Never leave the default at None in production.
StandardScaler for logistic / SVM / KNN.
log and runs faster on wide data.
Golden Rules — 4 to 6
✅ max_depth capped · ✅ random_state set · ✅ CV-tuned hyperparameters · ✅ Feature importance sanity-checked · ✅ Train-test gap under 5% · ✅ Ensemble considered.
Decision Trees — Simple, Readable, Foundational
You now understand tree anatomy, how impurity drives splits, why unpruned trees fail, and how feature importance falls out for free. Every tabular ML model you'll build from here — Random Forest, XGBoost, LightGBM, CatBoost — is a Decision Tree grown up.
Study Random Forest next (bagging + feature subsampling), then Gradient Boosting (XGBoost, LightGBM, CatBoost). Practise on Kaggle's Titanic or House Prices — watch a single tree lose to a properly-tuned forest by 3–5 points every time.
🌳 End of tutorial · Press ← to review, or click Restart