ROC Curve & AUC
Press Next → or use ← → arrow keys
What is ROC & AUC?
She needs a metric that captures classifier quality across every possible threshold, so she can pick the model with the best overall discrimination — then tune the threshold to her operational reality afterwards. That metric is ROC-AUC.
The ROC Curve plots True Positive Rate (y) vs False Positive Rate (x) as the decision threshold sweeps from 1 down to 0. The AUC is the area beneath that curve — a single number capturing overall separability.
The Two Rates That Build The Curve
TPR uses only the fraud row (P). FPR uses only the legit row (N). Each rate is normalised inside its own class — that's what makes the ROC curve robust to shifts in prevalence.
The Threshold Sweep — What Actually Moves
Slide τ left → catch more fraud (TPR ↑) but flag more legit transactions (FPR ↑). Slide τ right → fewer false alarms (FPR ↓) but miss more fraud (TPR ↓). The ROC curve traces this exact trade-off across every possible τ.
Building The Curve — Step By Step
Sort predictions by descending score. Sweep the threshold from high to low. Each time a true positive gets added, the curve steps up. Each false positive steps it right. Perfect classifier: all steps up first, then all steps right.
What AUC Actually Means
| AUC Range | Meaning | What To Do |
|---|---|---|
| 0.50 | Random guessing | Model is broken — rebuild features or targets |
| 0.50 – 0.70 | Poor discrimination | Needs work — check features, class balance |
| 0.70 – 0.90 | Good to very good | Production-ready in most domains |
| 0.90 – 1.00 | Excellent | Audit for data leakage before shipping |
AUC > 0.98 on a real-world problem usually means data leakage — a target-derived feature snuck into the training set, or your test set overlaps with training. Audit before celebrating.
Comparing Models — One Chart, Instant Verdict
The model whose curve hugs the top-left corner dominates at every threshold. Aisha picks Gradient Boost (AUC 0.93) — then tunes the operating point to her fraud team's capacity.
When ROC Lies — Meet PR-AUC
| Scenario | Use ROC-AUC | Use PR-AUC |
|---|---|---|
| Balanced classes (~50/50) | Best | Also works |
| Severe imbalance (< 1% positive) | Can mislead | Preferred |
| Fraud / rare-disease / spam | With caution | Much more informative |
FPR uses TN in its denominator. With 99,950 legit transactions, even hundreds of false positives leave FPR near zero — the ROC curve stays hugging the top-left corner while your analyst team drowns in false alarms. Precision ignores TN entirely — that's why PR-AUC tells the truth.
Picking The Operating Point
AUC scores the model. It does not pick your threshold — that's a business decision, not a maths one.
Selecting τ using test-set metrics is data snooping. Use a validation set (or nested cross-validation), then report final performance on the pristine test set.
Beyond Binary — Multi-Class ROC
Macro asks: "How well does the model discriminate the average class?" Weighted asks: "How well does it perform on the majority of samples?" Rare classes hide in the weighted average — always report both.
Implementation — From Scratch & sklearn
# ── FROM SCRATCH · sweep threshold, plot points ── def roc_curve_manual(y_true, y_scores): thresholds = np.sort(np.unique(y_scores))[::-1] P, N = np.sum(y_true == 1), np.sum(y_true == 0) tpr, fpr = [0.0], [0.0] for t in thresholds: y_pred = (y_scores >= t).astype(int) TP = np.sum((y_pred == 1) & (y_true == 1)) FP = np.sum((y_pred == 1) & (y_true == 0)) tpr.append(TP / P); fpr.append(FP / N) return np.array(fpr), np.array(tpr) auc = np.trapz(tpr, fpr) # trapezoidal rule # ── PRODUCTION · scikit-learn one-liner ───────── from sklearn.metrics import roc_curve, roc_auc_score y_proba = model.predict_proba(X_test)[:, 1] fpr, tpr, thr = roc_curve(y_test, y_proba) auc = roc_auc_score(y_test, y_proba) # ── Pick threshold via Youden's J ──────────────── best_idx = np.argmax(tpr - fpr) best_thresh = thr[best_idx]
A single AUC number can vary wildly on small test sets. Resample the test set 1,000× with replacement and report the 95% confidence interval — e.g. "AUC = 0.89 [0.85–0.93]". Two models whose CIs overlap are statistically indistinguishable.
Where ROC-AUC Earns Its Keep
Common Pitfalls — What Bites Beginners
Golden Rules — 1 to 3
Golden Rules — 4 to 6
✅ PR-AUC also computed · ✅ Bootstrap CI reported · ✅ Threshold picked on validation ✅ Cross-validated · ✅ Calibration inspected · ✅ Positive class verified.
ROC & AUC — Ranking Quality, Distilled
You now understand how the ROC curve is built, what AUC really measures, when it lies, and how to pick your operating point. Every classifier evaluation you do — logistic regression, XGBoost, neural networks — will rely on these ideas.
Study calibration curves & Platt scaling, then PR-AUC and Average Precision in depth, then cost-sensitive learning. Practice by comparing three models on Kaggle's Credit Card Fraud dataset — exactly Aisha's problem at scale.
📈 End of tutorial · Press ← to review, or click Restart