K-Nearest Neighbors
Press Next → or use ← → arrow keys
What Is K-Nearest Neighbors?
K-Nearest Neighbors does exactly this. It stores every training example, and to classify a new point it finds the K most similar stored points and lets them vote. There's no "training" in the usual sense — the data is the model.
KNN is an instance-based, lazy learner: it memorises the whole training set and defers all computation to prediction time. To classify, it finds the K nearest points and takes a majority vote; to regress, it takes their average.
How KNN Works — Four Steps
fit() just stores the data. All the work happens at predict() — fast to fit, slow to query.The K-Neighbourhood — Who Gets To Vote?
K controls how wide the neighbourhood circle is. A small K listens only to the very closest points (sensitive to noise); a large K polls a broad crowd (smooths everything out). Here K=5 captures 3 green and 2 red — the new point is classified green.
Choosing K — The Goldilocks Problem
| K value | Boundary | Behaviour | Risk |
|---|---|---|---|
| K = 1 | Jagged · noisy | Memorises each point | Overfits |
| K = 3–5 | Locally flexible | Respects local structure | Good for small clean data |
| K = 11–21 | Smooth | Well-generalised | Usually optimal |
| K = N | Flat | Always majority class | Underfits completely |
Test odd values from 1 to √n, plot CV accuracy vs K, and pick the elbow where accuracy peaks and flattens. Use odd K for binary problems to avoid tie votes.
Distance Metrics — How "Near" Is Measured
Feature Scaling — Mandatory, Not Optional
Age differs by ~1 (distance² = 1). Income differs by ₹40,000 (distance² = 1.6 billion). Without scaling,
income silently swamps every distance and age becomes invisible. Use
StandardScaler (or RobustScaler for outlier-heavy data) inside a
Pipeline to prevent leakage.
Weighted Voting — Let Closer Neighbours Speak Louder
Plain KNN gives all K neighbours an equal vote — even the one sitting right on the edge of the ring. Distance-weighting fixes that: the nearer a neighbour, the louder its vote.
Distance-weighted voting is especially valuable when the K-th neighbour is much farther than the
first — you don't want a distant, barely-relevant point overruling a very close one. Start with
weights='distance'; only switch to uniform if validation says otherwise.
The Curse Of Dimensionality
As features multiply, the distance to your nearest point and your farthest point
converge — so "nearest neighbour" carries no signal. Mitigate: apply
PCA (or UMAP) before KNN when features exceed 20–30, and prune irrelevant
features with Random-Forest importance or SelectKBest.
Making Prediction Fast — Search Structures
Every prediction compares the query to every stored point. On big data that's crippling — so sklearn offers smarter search than brute force.
algorithm='auto' lets sklearn choose based on data size, dimensionality, and metric.
For datasets beyond ~100k points where exact search is too slow, switch to
Approximate Nearest Neighbour libraries — FAISS (Meta),
Annoy (Spotify), ScaNN (Google) — for millisecond queries at billion scale.
Implementation — Sklearn, Pipeline & Best-K Search
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline from sklearn.model_selection import cross_val_score # ── Classification · always scale inside a pipeline ────── knn = Pipeline([ ('scaler', StandardScaler()), ('knn', KNeighborsClassifier( n_neighbors=5, weights='distance', # closer = louder metric='euclidean', n_jobs=-1, )), ]) knn.fit(X_train, y_train) # ── Find the best K with cross-validation ──────────────── best_k, best_score = 1, 0 for k in range(1, 32, 2): # odd values only pipe = Pipeline([ ('scaler', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=k)), ]) score = cross_val_score(pipe, X_train, y_train, cv=5).mean() if score > best_score: best_k, best_score = k, score print(f"best K = {best_k} ({best_score:.3f})") # ── Regression · distance-weighted mean ────────────────── reg = Pipeline([ ('scaler', StandardScaler()), ('knn', KNeighborsRegressor(n_neighbors=10, weights='distance')), ]).fit(X_train, y_train)
Fit StandardScaler only on each CV fold's training partition — that's what the
Pipeline guarantees. Scale the whole dataset up front and test-set statistics leak into
training, quietly inflating your reported accuracy.
Strengths & Weaknesses
Use it for low-dimensional data (< 20–30 features), small clean datasets, quick baselines, and explainability-critical problems. Avoid it for large-scale (> 100k), high-dimensional (text, images, genomics), or low-latency real-time inference — reach for SVM, tree ensembles, or ANN libraries instead.
Where KNN Shines In The Real World
Common Pitfalls — What Silently Sinks KNN
StandardScaler inside a Pipeline.weights='distance' beats uniform in almost every case. Make it your default.Golden Rules — Six Habits For KNN
StandardScaler (or RobustScaler for outliers) inside a
Pipeline. This single step is the most impactful thing you can do for KNN.
weights='distance'.
Closer neighbours are genuinely more relevant. Distance weighting almost always wins at near-zero cost.
KNN — Simplicity That Still Earns Its Keep
You now understand instance-based learning, the four-step vote, how K trades bias for variance, why scaling is non-negotiable, the distance metrics, and how the curse of dimensionality breaks "nearness". KNN is the most intuitive classifier in ML — and still a strong baseline on the right data.
Study PCA (to rescue KNN in high dimensions), then SVM and Random Forest for when KNN's cost or dimensionality becomes a problem. For scale, explore FAISS and approximate nearest-neighbour search. Practise on Iris, MNIST, and the Wine dataset.
📍 End of tutorial · Press ← to review, or click Restart