Logistic Regression
Press Next → or use ← → arrow keys
What is Logistic Regression?
Logistic Regression is the classifier that outputs exactly that — a probability between 0 and 1, then applies a threshold to turn it into a decision. Despite the name, it's classification, not regression.
Logistic Regression is a supervised algorithm for binary classification that predicts the probability an observation belongs to Class 1, then applies a threshold (default 0.5) to produce a label.
Why Linear Regression Fails at Classification
Feeding a linear model a 0/1 target technically runs — but the outputs are nonsense for a classifier.
Logistic Regression takes the same linear score and passes it through the sigmoid function. Every real number in, a valid probability out. Meet it on the next slide.
The Sigmoid — Any Number → A Probability
σ(0) = 0.5 (pure uncertainty). σ(+∞) → 1. σ(−∞) → 0. The curve is symmetric around zero. Any linear score enters; a valid probability comes out.
The 4-Step Pipeline
"Malignant" is a decision. "87% confident this is malignant" is a usable decision. Send both to the human downstream.
Log-Odds — Where The Name Comes From
Invert the sigmoid algebraically and something beautiful appears: the log-odds of the outcome are a linear function of the features.
| Probability | Odds p / (1 − p) | Log-Odds ln(odds) | Meaning |
|---|---|---|---|
| 0.10 | 0.111 | −2.20 | Strongly Class 0 |
| 0.25 | 0.333 | −1.10 | Likely Class 0 |
| 0.50 | 1.000 | 0.00 | Coin flip |
| 0.75 | 3.000 | +1.10 | Likely Class 1 |
| 0.90 | 9.000 | +2.20 | Strongly Class 1 |
Log-odds are unbounded (−∞ to +∞) — the natural output space of a linear model. The sigmoid is just a translation layer between the linear world and the probability world.
The Cost Function — Binary Cross-Entropy
Log loss explodes toward infinity for confidently-wrong predictions. That's what forces the model to output well-calibrated probabilities rather than lazy 50/50 guesses.
Gradient Descent — One Beautiful Update Rule
A single-iteration walk-through — 4 samples, initial β = 0, learning rate α = 0.1:
| Size (x) | Label (y) | z = 0 | p̂ = σ(0) | Error (p̂ − y) |
|---|---|---|---|---|
| 2 | 0 | 0 | 0.5 | +0.5 |
| 3 | 0 | 0 | 0.5 | +0.5 |
| 5 | 1 | 0 | 0.5 | −0.5 |
| 7 | 1 | 0 | 0.5 | −0.5 |
∂J/∂β₁ = (0.5·2 + 0.5·3 − 0.5·5 − 0.5·7) / 4 = −0.875. β₁ ← 0 − 0.1·(−0.875) = +0.0875. After just one step, larger tumours already receive higher malignancy probability. Convex loss → guaranteed global minimum. ✅
The Decision Boundary — A Straight Line in 2D
A line in 2D, a plane in 3D, a hyperplane in n-D. If your classes are non-linearly separable (one wrapped around the other), plain logistic regression will struggle — reach for polynomial features, kernels or tree-based models.
Confusion Matrix — The Truth Table
A False Positive means an unnecessary biopsy — inconvenient. A False Negative means a missed cancer — potentially fatal. In medicine you tune the threshold to minimise FN even at the cost of extra FP.
Accuracy · Precision · Recall · F1
Drop Dr. Sharma's threshold from 0.5 → 0.3 and recall jumps from 92% to 97% (fewer missed cancers). The trade-off: precision falls to 74% (more benign biopsies). In oncology, that trade is worth it. In spam-filtering, the reverse.
ROC Curve & AUC — Threshold-Independent Quality
| AUC Range | Verdict | What It Means |
|---|---|---|
| 0.50 | Random | Model has no discriminative power |
| 0.70 – 0.80 | Acceptable | Useful for triage / ranking |
| 0.80 – 0.90 | Good | Production-ready in most domains |
| 0.90 – 1.00 | Excellent | Check for data leakage before celebrating |
AUC = the probability the model ranks a random positive higher than a random negative. It's threshold-independent — perfect for comparing model versions. For very imbalanced data (fraud < 0.1%), use PR-AUC instead.
Regularisation & Multiclass Extensions
One-vs-Rest (OvR) trains K binary classifiers, one per class — simple & fast. Softmax (Multinomial) trains one model with K outputs; the softmax function turns scores into a proper probability distribution that sums to 1. Softmax is the industry default and the foundation of neural-network classifiers.
Implementation — Scratch vs Scikit-learn
# ── FROM SCRATCH · the whole model in ~20 lines ── import numpy as np def sigmoid(z): return 1 / (1 + np.exp(-z)) def train_logistic(X, y, lr=0.1, epochs=1000): n, p = X.shape beta = np.zeros(p + 1) X_b = np.column_stack([np.ones(n), X]) for _ in range(epochs): p_hat = sigmoid(X_b @ beta) grad = (1/n) * (X_b.T @ (p_hat - y)) beta -= lr * grad return beta # ── PRODUCTION · scikit-learn one-liner ───────── from sklearn.linear_model import LogisticRegression from sklearn.preprocessing import StandardScaler X_train = StandardScaler().fit_transform(X_train) # ALWAYS scale! model = LogisticRegression(C=1.0, max_iter=1000) model.fit(X_train, y_train) proba = model.predict_proba(X_test)[:, 1] # Class 1 probability
scaler.fit_transform(X_train) then scaler.transform(X_test) —
never re-fit. Fitting on the full dataset leaks test statistics into training and inflates every metric.
Common Pitfalls — What Bites Beginners
StandardScaler.Golden Rules — 1 to 3
StandardScaler. Fit on training only,
transform test — anything else is data leakage.
Golden Rules — 4 to 6
C=1.0 (L2) is a safe start. Try L1 with many noisy features. Always tune C via cross-validation.
A properly tuned logistic-regression baseline is often the model in production for years. Fancy techniques earn deployment only when they beat this baseline meaningfully.
Logistic Regression — Probabilities, Not Guesses
You now understand the sigmoid, log-odds, log-loss, gradient descent, decision boundaries, confusion matrices and ROC-AUC. Every classifier that follows — SVMs, random forests, gradient boosting, even the softmax layer of a neural network — extends the ideas you just learned.
Study softmax & multiclass classification, then Naive Bayes for comparison, then jump to SVMs and tree-based classifiers. Practice on Kaggle's Titanic and Breast Cancer datasets — exactly Dr. Sharma's problem.
🎯 End of tutorial · Press ← to review, or click Restart