Support Vector Machines
Press Next → or use ← → arrow keys
What Is A Support Vector Machine?
A Support Vector Machine does exactly this. It hunts for the separating boundary that maximises the empty gap (the margin) between the two classes. A wide margin means more confidence — and better generalisation to new data.
SVM is a supervised classifier that finds the maximum-margin hyperplane — the boundary with the largest possible distance from the nearest points of each class. Those nearest points are called support vectors, and they alone determine the whole model.
The Maximum-Margin Hyperplane
Only the points touching the two margin lines matter — they are the support vectors. Delete every other training point (98% of them, sometimes) and retrain — you get the exact same model. That's why SVMs are so memory-efficient at inference time.
The Math In Four Lines
Only the points that sit exactly on the margin lines (or violate them with a slack ξᵢ > 0) influence w and b. Every other point contributes zero to the solution — hence "support" vectors: the tiny subset of the data that holds the whole hyperplane up.
Hard Margin vs Soft Margin — The C Trade-Off
| C value | Behaviour | Symptom |
|---|---|---|
| C = 100 | Narrow margin · zero tolerance | Overfits · one noisy point moves the line |
| C = 1.0 | Balanced · few slack violations allowed | Usually the sweet spot |
| C = 0.001 | Very wide margin · many violations OK | Underfits · misses real patterns |
Real data is messy — perfect separation isn't feasible. Soft-margin SVM lets a few points cross the boundary by paying a slack cost. C sets the price. High C → strict. Low C → forgiving. Always tune on a log grid.
The Kernel Trick — Curved Boundaries, Linear Math
Actually mapping every point into a high-dimensional space would be prohibitively expensive. The kernel trick uses a clever identity — K(xᵢ, xⱼ) = φ(xᵢ) · φ(xⱼ) — that computes the dot product as if the points had been lifted, without ever performing the lift. Non-linear boundaries at linear-model cost.
The Four Kernels You'll Use
| Kernel | Best For | Watch Out For |
|---|---|---|
| Linear | Text, TF-IDF, wide sparse features | Straight lines only — can't curve |
| RBF (default) | Any non-linear problem | Sensitive to γ — needs tuning |
| Polynomial | Feature interactions, image data | Overflows on high degrees |
| Sigmoid | Rarely a first choice | Unstable · often doesn't converge |
The γ Parameter — Reach Of Each Support Vector
Always use gamma='scale' as your default (it adapts to feature variance).
Never tune C alone — the two parameters push and pull each other. Sweep both on a log grid
{0.001, 0.01, 0.1, 1, 10, 100} via GridSearchCV and let CV pick the winner.
Hinge Loss — Only Punish The Wrong Side
Beyond Binary — SVR & Multi-Class
SVC default for multi-class.Strengths & Weaknesses
SGDClassifier(loss='hinge') instead.StandardScaler lets one large-scale feature dominate every margin.probability=True for Platt-scaled probabilities — slower to train.Logistic Regression is faster and gives calibrated probabilities — often the safer default. Random Forest handles feature interactions natively and doesn't need scaling. SVM wins on wide sparse data (text) and when the margin idea genuinely helps generalisation. For under 10k samples with a clean signal, SVM with RBF still holds its own.
Implementation — Sklearn In A Handful Of Lines
from sklearn.svm import SVC, LinearSVC, SVR from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline from sklearn.model_selection import GridSearchCV import numpy as np # ── Classification with RBF · always scale + pipeline ───────── svm = Pipeline([ ('scaler', StandardScaler()), ('clf', SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)), ]) svm.fit(X_train, y_train) # ── Tune C AND γ together on a log grid ────────────────────── grid = GridSearchCV(svm, param_grid={ 'clf__C': np.logspace(-3, 3, 7), 'clf__gamma': np.logspace(-4, 2, 7), }, cv=5, scoring='roc_auc', n_jobs=-1).fit(X_train, y_train) print(grid.best_params_, grid.best_score_) # ── Text · LinearSVC scales far better than kernel SVC ─────── text = Pipeline([ ('tfidf', TfidfVectorizer(ngram_range=(1, 2))), ('clf', LinearSVC(C=1.0, max_iter=5000)), ]).fit(docs_train, y_train) # ── Regression with ε-tube ─────────────────────────────────── svr = Pipeline([ ('scaler', StandardScaler()), ('svr', SVR(kernel='rbf', C=100, epsilon=0.1)), ]).fit(X_train, y_train)
Always wrap in a Pipeline so the scaler refits per CV fold — no data leakage.
Set probability=True at construction (not later) if you need calibrated probabilities;
switching it on afterward means Platt scaling doesn't happen and predict_proba disagrees
with predict.
Where SVM Still Wins In Production
Common Pitfalls — What Silently Breaks SVM
StandardScaler, one large-scale feature swamps every margin calculation.GridSearchCV.SVC trains for hours. Use LinearSVC or SGDClassifier(loss='hinge').predict_proba gives inconsistent answers.Golden Rules — Seven Habits For SVM Practitioners
StandardScaler inside a Pipeline. SVM is distance-based;
skipping this step silently destroys the margin geometry.
{0.001, 0.01, 0.1, 1, 10, 100} for both via GridSearchCV.
They push against each other — one at a time misses the interaction.
gamma='scale'.
It adapts to feature variance automatically. Only override if you're deliberately searching a grid.
probability=True at construction time.
Enables Platt scaling for calibrated probabilities. Skipping it means decision_function
scores are all you get — useful, but not probabilities.
SVM — The Elegant Classifier That Refuses To Retire
You now understand maximum-margin classification, why only support vectors matter, how the C parameter trades margin width for accuracy, how the kernel trick delivers non-linear boundaries at linear-model cost, and how γ shapes the RBF reach. Every SVM you tune from now on is a variation on these ideas.
Study Kernel PCA (same kernel trick, unsupervised), then One-Class SVM & Isolation Forest for anomaly detection, then SGDClassifier(loss='hinge') for SVM at scale. Practise on the MNIST, 20 Newsgroups and Breast Cancer datasets — SVM excels on all three.
🛡️ End of tutorial · Press ← to review, or click Restart