📈

Linear Regression

The straight line that turned into the workhorse of predictive analytics — its equation, assumptions, cost function, evaluation metrics and the pitfalls every practitioner must know.

The Best-Fit Line OLS & Cost Function 5 Key Assumptions R² · RMSE · MAE

Press Next → or use ← → arrow keys

Section 01

What is Linear Regression?

🏢 REAL-WORLD ANALOGY

Priya, a Mumbai real-estate analyst, needs to price a flat

Priya has data on flats she has already sold — size in sq ft and price in lakhs. Tomorrow a client walks in with a 1,200 sq ft flat. She needs a defensible answer, not a guess.

Linear Regression gives Priya a formula: feed in the size, out comes the price. It finds the single straight line that best explains the relationship between her input (size) and her output (price) — and then uses that line to predict the future.

💡

The Working Definition

Linear Regression finds the best-fit straight line that describes the relationship between one or more input features and a continuous target — then uses that line to predict new values.

1Straight line, best fit

2Types — simple & multiple

5Assumptions that must hold

200+Years old — still winning

Section 02

The Best-Fit Line — In Pictures

📐

Two Numbers Define The Whole Model

The slope (β₁) says how much price changes per extra sq ft. The intercept (β₀) says where the line crosses the y-axis. That's it — the entire model is just two numbers.

Section 03

The Equation — Simple & Multiple

Simple Linear Regression

ŷ = β₀ + β₁ · x

One feature. One slope. One intercept. Priya's size → price model.

Multiple Linear Regression

ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ

Many features (size, location, floor, age). Each βᵢ = that feature's impact.

📍

β₀ — Intercept

where line meets y-axis

The predicted value of y when every feature is zero. A baseline anchor for the line.

📈

β₁ — Slope

rate of change

How much y changes for every 1-unit increase in x. Priya's β₁ = 0.0975 → ₹9,750 per sq ft.

🎯

ŷ — Prediction

the model's output

The model's estimate. Reality (y) may still differ — the gap is the residual we're about to meet.

🧮

Interpretability Is Regression's Superpower

Every coefficient in a linear model has a plain-English meaning: hold everything else constant, a 1-unit change in xᵢ moves the prediction by βᵢ. That's why regulators, banks and doctors still trust it decades after fancier models arrived.

Section 04

Residuals — The Gaps We Try To Shrink

🎯

The Objective — Minimise The Sum of Squared Gaps

A residual is the vertical gap between an actual point and the model's line. We square each residual (positive gaps, big-error penalty), add them up, and then find the line that makes that total as small as possible. That total is the cost function.

Section 05

Cost Function & The Descent To The Best Line

Cost — Sum of Squared Residuals

L = Σ (yᵢ − ŷᵢ)²

The total squared error. Squaring makes gaps positive & punishes big misses harder.

OLS Closed-Form (Slope)

β₁ = Cov(x, y) / Var(x)

And β₀ = ȳ − β₁ · x̄. One-shot, no iteration needed — the optimal answer exists in a formula.

⛰️

Two Ways To Reach The Bottom

For small datasets, OLS closed-form jumps to the minimum in one calculation. For massive datasets, gradient descent rolls down the bowl step by step — slower per iteration, but memory-friendly and the only option once your data won't fit in RAM.

Section 06 · Worked Example

Priya's Flats — From Data To Prediction

Size (sq ft)	Price (₹ L)	Predicted ŷ	Residual (y − ŷ)
600	65	65.0	0.0
800	85	84.5	+0.5
1,000	105	104.0	+1.0
1,200	120	123.5	−3.5
1,400	145	143.0	+2.0

Slope (β₁)

39,000 / 400,000 = 0.0975

Every extra sq ft adds ≈ ₹9,750 to the price.

Intercept (β₀)

104 − (0.0975 × 1,000) = 6.5

The line crosses the y-axis at ₹6.5 L (extrapolation only — not a real 0 sq ft flat!).

🏠

The Model & A Live Prediction

ŷ = 6.5 + 0.0975 · x. For a 1,200 sq ft flat: ŷ = 6.5 + 0.0975 × 1,200 = ₹123.5 lakhs. Priya has a defensible number for her client.

Section 07

R² — How Much Better Than "Just The Average"?

📊

R² = 1 − (SSᵣₑₛ / SSₜₒₜ)

R² measures the fraction of variance your regression explains beyond the naïve "just guess the mean" baseline. Priya's model: R² = 1 − 17.5 / 3,820 = 0.9954 — it explains 99.5% of price variance. That's excellent for 5 clean data points.

Section 07 · Metrics

Evaluation Metrics — Speak In The Right Units

RMSE — Root Mean Squared Error

√( Σ(yᵢ − ŷᵢ)² / n )

Same unit as your target. Priya's RMSE = ₹1.87 L — a number her boss instantly understands.

MAE — Mean Absolute Error

Σ |yᵢ − ŷᵢ| / n

Treats every error equally. Robust to outliers where RMSE would over-react.

R² — Coefficient of Determination

1 − SSᵣₑₛ / SSₜₒₜ

Variance explained (0 to 1). Warning: R² almost always rises when you add features.

Adjusted R²

1 − (1 − R²)(n − 1) / (n − p − 1)

Penalises useless features. The right metric when comparing models of different sizes.

R² Range	Verdict	Typical Domain
0.90 – 1.00	Excellent	Clean physics, controlled experiments
0.70 – 0.90	Good	Housing, sales forecasting
0.50 – 0.70	Acceptable	Marketing response, human behaviour
0.00 – 0.50	Poor	Financial returns, noisy signals
Negative	Worse than the mean	Model is broken — rebuild

Section 08

The Five Assumptions — Break One, Break The Model

📏

Linearity

The x–y relationship is actually a line. Check: scatter plot before anything else.

🔗

Independence of Errors

Residuals don't correlate with each other. Check: Durbin-Watson test (especially for time-series).

📊

Homoscedasticity

Constant residual spread across all x. Check: residuals-vs-fitted plot — no funnel shape.

🔔

Normal Residuals

Errors follow a bell curve — needed for reliable confidence intervals. Check: Q-Q plot.

🚫

No Multicollinearity

Features aren't strongly correlated with each other. Check: VIF ideally < 5.

⚠️

No Influential Outliers

One extreme point can twist the whole line. Check: Cook's Distance.

🎯

Gauss–Markov In One Line

When these assumptions hold, OLS gives you the Best Linear Unbiased Estimator (BLUE) — no other linear method can produce lower variance. Break the assumptions, and you lose that guarantee.

Section 09

Implementation — Scratch vs Scikit-learn

Two ways to fit the same line: understand the mechanics from scratch, then let a battle-tested library handle the edge cases in production.

# ── OLS from scratch ──────────────────────────
x_bar = sum(x) / n
y_bar = sum(y) / n

beta1 = (sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y))
         / sum((xi - x_bar) ** 2 for xi in x))
beta0 = y_bar - beta1 * x_bar

# ── The same, with scikit-learn ───────────────
from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X, y)
y_pred = model.predict(X)

print(model.coef_, model.intercept_)

🧪

Build It Once — Then Never Again

Writing OLS from scratch teaches you what the model is doing. After that, always reach for sklearn.linear_model.LinearRegression or statsmodels.OLS — they handle numerical stability, missing values and diagnostics for free.

Section 10

Common Pitfalls — What Bites Beginners

📈

R² Always Rises

the phantom improvement

Adding any feature — even pure random noise — can only increase plain R². Use Adjusted R² or a validation split.

🎯

Outlier Sensitivity

one bad point rules them all

Squaring residuals makes one billionaire's flat drag the whole line. Inspect residuals; consider robust regression when needed.

🔀

Broken Assumptions

silent invalidation

Curved data, changing variance, correlated errors, high VIF — the model still runs, but its p-values and intervals become lies.

🔍

Train ≠ Test

the overfitting tell

Training R² = 0.99, test R² = 0.55? You've memorised, not learned. Always evaluate on held-out data.

🧭

Extrapolation Danger

outside the training range

Priya's model was trained on 600–1,400 sq ft. Predicting a 5,000 sq ft penthouse is fantasy — the line has never seen that region.

🤝

Correlation ≠ Causation

the classic warning

A significant coefficient tells you two things move together. It does not prove one causes the other.

Section 11

Where Linear Regression Still Wins

🏠

Real Estate

Property valuation from size, location, age, floor, amenities — Priya's exact use case.

💰

Finance

CAPM asset-pricing, credit scoring, salary benchmarking, expense forecasting.

📢

Marketing

Ad-spend attribution, sales lift modelling, customer-lifetime-value estimation.

🏥

Healthcare

Dosage prediction, BMI vs blood pressure, hospital-stay length estimation.

🌾

Agriculture & Climate

Crop yield from rainfall, fertiliser and soil quality — the field where OLS was born.

⚙️

Engineering & QC

Material strength vs composition, energy demand forecasting, calibration curves.

🥇

Why It Still Matters

Linear regression is the baseline every real ML project starts from. If a neural network can't beat a properly tuned linear model by a meaningful margin, the complexity isn't earning its keep — ship the line.

Section 12 · Part 1

Golden Rules — 1 to 5

📈 LINEAR REGRESSION DISCIPLINE · RULES 1–5

1

Always plot first. A scatter plot instantly tells you whether a straight line is even the right tool. Fitting a line to curved data is not a bug — it's malpractice.

2

Report RMSE in the target's units. "RMSE = ₹1.87 lakhs" beats "MSE = 3.50" every time. Business owners think in units, not squared units.

3

Use Adjusted R² for model comparison. Plain R² almost never decreases when you add features — even garbage ones. Adjusted R² punishes bloat.

4

Check every assumption before you trust the p-values. Residual plot, Q-Q plot, VIF, Durbin-Watson. Skipping diagnostics is skipping the science.

5

High R² ≠ useful model. You can hit R² = 0.99 by memorising the training data. Always validate on held-out samples.

Section 12 · Part 2

Golden Rules — 6 to 10

📈 LINEAR REGRESSION DISCIPLINE · RULES 6–10

6

Scale features for gradient-descent solvers. Standardise to mean 0, std 1 before iterative methods. Faster convergence, no dominant-feature bias.

7

Never extrapolate beyond the training range. A line trained on 600–1,400 sq ft has zero knowledge of a 5,000 sq ft penthouse. Predict inside the box, not outside.

8

Inspect residuals, not just metrics. A residual-vs-fitted plot reveals patterns the summary numbers hide — curvature, funnel shapes, seasonality.

9

Consider regularisation (Ridge / Lasso). When features multiply or multicollinearity creeps in, penalised regression stabilises coefficients and reduces overfitting.

10

Ship the baseline first. A calibrated linear model in production beats a Kaggle-winning deep net still on your laptop.

FINAL

Linear Regression — Two Numbers, Endless Uses

β₀ + β₁xThe whole equation

5Assumptions to verify

4Core metrics

6Pitfalls to dodge

10Golden rules

99.5%Priya's R² · variance explained

🎯

The Foundation Is Set

You now understand the equation, the cost function, the assumptions, the metrics and the traps. Everything that follows — logistic regression, ridge, lasso, GLMs, even neural networks — is a variation on the theme you just learned.

📚

Where To Go Next

Learn Logistic Regression (classification cousin), then Ridge & Lasso (regularised variants), then Polynomial & GLMs for non-linear extensions. Practice on Kaggle's House Prices dataset — it's Priya's problem at scale.

📈 End of tutorial · Press ← to review, or click Restart