🧮

Entropy · Information Gain · Gini

The three numbers a decision tree uses to decide which question to ask first — Shannon's uncertainty in bits, the probability of misclassification, and the gain that picks the winner.

Entropy H(S) Gini 1−Σp² Information Gain Gain Ratio

Press Next → or use ← → arrow keys

Section 01

Impurity — The Foundation Of Every Split

🎲 REAL-WORLD ANALOGY

Guess the coin — 20 questions or 1?

If a bag has only heads coins, you don't need to ask anything — you know the answer. That's an impurity of zero. If it's 50/50 heads and tails, you need exactly 1 yes/no question — an impurity of one bit.

Impurity measures how mixed the classes are inside a group. A decision tree's entire job is to find splits that make the child groups less mixed than the parent. Entropy and Gini are the two ways to measure that mixing.

💡

Two Metrics, One Goal

Entropy counts the yes/no questions needed to guess a sample's class. Gini asks how often you'd mislabel a random sample. Both hit zero when a node is pure and peak when the classes are perfectly balanced.

H(S)Shannon entropy · bits

1−Σp²Gini impurity

IGInfo gain picks the split

0 = pureBoth metrics · zero perfect

Section 02

Entropy — Uncertainty Measured In Bits

🎯

Why Bits?

H = 1 means you need exactly one yes/no question to guess the class (like a coin flip). H = 0 means you need none — you already know. H = 3 (for 8 balanced classes) means you'd need 3 questions. That's Shannon's information theory in one sentence.

Section 03

Step-By-Step — Play Tennis Parent Entropy

The classic dataset: 14 days · 9 Play · 5 Don't Play. Before any split, how uncertain is the parent?

Step 1 · Proportions

p(Yes) = 9/14 = 0.643 · p(No) = 5/14 = 0.357

Count each class, divide by total.

Step 2 · Terms

−0.643·log₂(0.643) = 0.410

And −0.357·log₂(0.357) = 0.530. One term per class.

Step 3 · Sum

H(S) = 0.410 + 0.530 = 0.940 bits

The parent's entropy — the number every split will try to reduce.

Convention

0 · log₂(0) = 0

Add a tiny epsilon in code (+ 1e-12) to avoid NaN when a class has zero samples.

Classes n	Max Entropy log₂(n)	Example Domain
2	1.000 bits	Yes / No
3	1.585 bits	Sunny · Overcast · Rain
4	2.000 bits	Seasons
10	3.322 bits	Digit recognition 0–9

Section 04

The Impurity Spectrum — Pure To Chaotic

📊

Both Metrics Agree On The Extremes

Pure → zero. Balanced → maximum. Between them they curve slightly differently — but they always agree on the winner when the choice is obvious. That's why sklearn defaults to the faster Gini, and only true information-theory work justifies switching to Entropy.

Section 05

Entropy vs Gini vs Misclassification

Property	Entropy	Gini
Formula	−Σ pᵢ·log₂(pᵢ)	1 − Σ pᵢ²
Binary max	1.0 bit	0.5
n-class max	log₂(n)	(n−1)/n
Computation	slower (log)	faster (square)
Sensitivity at extremes	higher	smoother
sklearn	criterion='entropy'	default

Section 06

Gini — The Fast Cousin

Gini Impurity

Gini(S) = 1 − Σ pᵢ²

Range 0 (pure) to (n−1)/n. Peaks at 0.5 for binary. No logarithm — just squares and one subtract.

Play Tennis Parent

1 − (0.643² + 0.357²) = 0.460

Compare with Entropy's 0.940 bits — different numbers, identical judgement.

Intuition: if you pick a random sample from the node and randomly label it according to the node's class distribution, what's the probability you get it wrong? That probability is Gini.

Classes n	Max Gini (n−1)/n	Comparison to Entropy log₂(n)
2	0.500	Entropy: 1.000 bits
3	0.667	Entropy: 1.585 bits
4	0.750	Entropy: 2.000 bits
5	0.800	Entropy: 2.322 bits
10	0.900	Entropy: 3.322 bits

📐

Never Compare Raw Values Across Metrics

"Gini 0.460 vs Entropy 0.940" tells you nothing — different ranges. Only compare gain within the same metric. Both will nearly always pick the same split anyway.

Section 07

Information Gain — How A Split Is Scored

Information Gain

IG(S, A) = H(S) − Σ (|Sᵥ|/|S|) · H(Sᵥ)

Parent entropy minus the weighted average of children's entropy. Higher IG = better split.

Interpretation

IG ≥ 0 always

Splitting never increases expected entropy. IG = 0 means a useless split; IG = H(parent) means perfect purity.

Play Tennis — splitting on Outlook:

Child	[Yes, No]	Weight	Entropy	Contribution
Sunny	[2, 3]	5/14	0.971	0.347
Overcast	[4, 0]	4/14	0.000	0.000
Rain	[3, 2]	5/14	0.971	0.347
Weighted child entropy				0.694
Information Gain				0.940 − 0.694 = 0.246 bits

🏆

Overcast Is A Free Win

The Overcast child is perfectly pure — every one of the 4 Overcast days is a "Yes". That single pure child alone recovers 25% of the parent's entropy. This is why Outlook decisively beats the alternatives.

Section 08

The Winner — Information Gain Bar Chart

🎯

Zero-Gain Features Reveal Themselves

Temperature's 0.029 bit gain is so tiny it will never win a split anywhere in the tree. The feature-importance report will show it as zero. A powerful free feature-selection signal: drop it and retrain leaner.

Section 09

Good Split vs Bad Split — Visualised

⚖️

IG Is A Simple Question

"After this split, how much less uncertain am I?" A great split turns confusion into certainty (IG large). A pointless split preserves the confusion (IG = 0). The tree tries every split and keeps only the ones that meaningfully reduce IG.

Section 10

Gain Ratio — Fixing IG's Fatal Flaw

Raw Information Gain has a bias: it loves features with many unique values, even when they're useless. Add a "Day ID" column with 14 unique values, and it wins with IG = 0.940 — the maximum possible.

SplitInfo — the bias measure

SplitInfo(S, A) = − Σ (|Sᵥ|/|S|) · log₂(|Sᵥ|/|S|)

Entropy of the split itself. Many-way splits have high SplitInfo — a penalty term.

Gain Ratio (C4.5)

GainRatio = IG(S, A) / SplitInfo(S, A)

Normalises IG by split complexity. Levels the field between low- and high-cardinality features.

🎚️

Filter IDs, Timestamps & Free-Text Before Modelling

Any column with near-unique values per row (transaction IDs, timestamps, names) will hijack raw IG. Either drop them, bin them, or use Gain Ratio / Gini which are less susceptible to the bias.

Section 11

Implementation — Entropy, Gini & IG From Scratch

import numpy as np

def entropy(labels):
    """Shannon entropy of class labels."""
    n = len(labels)
    if n == 0: return 0.0
    _, counts = np.unique(labels, return_counts=True)
    probs = counts / n
    return -np.sum(probs * np.log2(probs + 1e-12))    # epsilon avoids NaN

def gini(labels):
    """Gini impurity of class labels."""
    n = len(labels)
    if n == 0: return 0.0
    _, counts = np.unique(labels, return_counts=True)
    probs = counts / n
    return 1 - np.sum(probs ** 2)

def information_gain(parent, children):
    """IG = H(parent) − weighted Σ H(child)."""
    n = len(parent)
    return entropy(parent) - sum(
        (len(c) / n) * entropy(c) for c in children
    )

# ── sklearn — swap the criterion in one word ────
from sklearn.tree import DecisionTreeClassifier
dt_gini = DecisionTreeClassifier(criterion='gini').fit(X, y)      # default, faster
dt_ent  = DecisionTreeClassifier(criterion='entropy').fit(X, y)   # slightly slower

🧪

Both Metrics Almost Always Agree

On clean datasets Gini and Entropy build identical or near-identical trees. Differences surface only on noisy, imbalanced data. Default to Gini for speed; switch to Entropy when you need extra sensitivity in the tail probabilities.

Section 12

When To Use Which Metric

⚡

Gini

CART · sklearn default

Fastest (no log). Best for balanced datasets, production systems, and most tabular problems.

📐

Entropy

ID3 · C4.5

More sensitive to skewed distributions. Preferred for imbalanced classification, research, information-theoretic work.

⚖️

Gain Ratio

C4.5 refinement

Use when features have wildly different cardinalities. Prevents IDs and timestamps from hijacking the split search.

🎯

A Practical Decision Tree For Choosing A Metric

Balanced classes? → Gini. Imbalanced (fraud, disease)? → Entropy. Features with hugely varied cardinality? → Gain Ratio. When in doubt, run both and pick the higher validation-set score.

Section 13

Common Pitfalls — What Silently Breaks Splits

🔢

High-Cardinality Hijack

IDs win everything

Raw IG loves unique-value columns. Filter IDs and timestamps, or use Gain Ratio / Gini.

🧮

0 · log₂(0) NaN

the epsilon trap

Mathematically 0·log₂(0) = 0 by convention. In code add + 1e-12 inside the log to avoid NaN.

📏

Mixing Metric Ranges

Gini 0.5 ≠ Entropy 0.5

Never compare raw Gini and Entropy numbers. Compare gain within the same metric.

📊

Imbalance Blindness

"low entropy" ≠ good

A 990/10 split has entropy ≈ 0.08 — looks pure, but those 10 fraud cases matter most. Use class_weight='balanced'.

🔀

Cross-Dataset IG

no absolute meaning

"IG = 0.4" on one dataset means nothing on another. Parent entropy differs. Use Gain Ratio for cross-comparisons.

🚫

Chasing 0 Gain

exhaustive search waste

IG = 0 means the split is useless — the tree won't take it. Save time; stop searching when max IG is below threshold.

Section 14

Golden Rules — Six Habits For Impurity Metrics

🧮 ENTROPY · GINI · IG DISCIPLINE

1

Entropy and Gini measure the same idea — pick by speed or theory. Both hit zero at purity and peak at balance. Gini avoids the log; Entropy has richer math. Trees built with either are nearly identical.

2

Information Gain is always ≥ 0. Splitting can never increase expected entropy. IG = 0 signals a useless split — the tree ignores it.

3

Raw IG is biased toward high-cardinality features. IDs, timestamps and near-unique columns will always win. Reach for Gain Ratio (C4.5) or use Gini — it's less susceptible.

4

Max impurity scales with class count. Entropy max = log₂(n). Gini max = (n−1)/n. Never compare raw values across problems of different complexity.

5

Impurity is training-time only. During prediction, the tree just follows branches — zero inference cost. Entropy and Gini optimise the search, not runtime behaviour.

6

Zero feature importance = the tree told you it's noise. If a feature never wins the max-gain race at any node, drop it. Retrain leaner and faster with no accuracy loss.

FINAL

Impurity — The Compass Every Tree Follows

H(S)Entropy · bits

1−Σp²Gini · probability

IGPicks the winning split

GRFixes cardinality bias

6Pitfalls to dodge

6Golden rules

🎯

The Foundation Is Set

You now understand what impurity means, why Entropy and Gini both work, how Information Gain picks the winner, and why Gain Ratio saves you from ID hijacks. Every tree-based model — Decision Tree, Random Forest, XGBoost, LightGBM — runs on these three numbers.

📚

Where To Go Next

Study Decision Trees in depth (pruning, hyperparameters), then Random Forest (bagging), then Gradient Boosting (XGBoost / LightGBM). Practise by tracing IG calculations on Kaggle's Titanic dataset — you'll see feature importance drop out for free.

🧮 End of tutorial · Press ← to review, or click Restart