✨

SVM Kernels

The mathematical shortcut that lets a linear classifier curve through non-linear data — Linear, RBF, Polynomial, Sigmoid — and how to pick and tune the right one for your problem.

The Kernel Trick 4 Kernel Types C & γ Tuning Grid Search

Press Next → or use ← → arrow keys

Section 01

What Is A Kernel, And Why Do We Need One?

🎈 REAL-WORLD ANALOGY

Red dots trapped inside blue dots on a table

Picture a flat table. Red dots cluster in a circle at the centre; blue dots form a ring around them. No matter how you tilt a straight ruler, you can't separate the two colours with a single line — the pattern isn't linearly separable.

Now imagine gently pushing the red dots upward based on how close they sit to the middle. Suddenly you have a 3D scene: reds float high, blues stay flat. A horizontal card can slice cleanly between them. That upward push is what a kernel does — invisibly, and without ever actually moving the points.

💡

The Working Definition

A kernel function K(a, b) measures how similar two data points are, as if they had been lifted into a much richer feature space — without ever computing the coordinates in that space. This trick lets a linear SVM discover curved, circular, or wave-shaped decision boundaries at the cost of a plain dot product.

K(a,b)Similarity function

φ(x)The lift · never computed

4Common kernels in sklearn

∞Dimensions RBF implicitly uses

Section 02

The Kernel Trick — Visualised

✨

The Beautiful Shortcut

SVM training and prediction only ever need dot products between pairs of points. A kernel function computes that dot product as if the points had already been lifted — no coordinates in the lifted space are ever built. You get non-linear boundaries at the cost of a simple similarity calculation.

Section 03

The Four Kernels You'll Actually Use

Linear · plain dot product

K(a, b) = aᵀ · b

No transformation. Straight-line boundaries. Best for text, TF-IDF, wide sparse data.

RBF · Gaussian · the default

K(a, b) = exp(−γ · ‖a − b‖²)

Similarity that decays with distance. Infinite implicit dimensions. Smooth curved boundaries.

Polynomial · feature interactions

K(a, b) = (γ · aᵀb + r)ᵈ

Captures products up to degree d. Waves and parabolas. Good for vision and NLP.

Sigmoid · rarely used

K(a, b) = tanh(γ · aᵀb + r)

Neural-net-like curves. Often not a valid kernel · use actual neural nets instead.

🎯

One Line To Remember

Linear for text · RBF for the unknown · Polynomial when features interact meaningfully · Sigmoid almost never — reach for a neural network instead.

Section 04

Decision Boundaries — Same Data, Four Kernels

📏

Match The Boundary Shape To Your Data

If your two classes are separated by a curve, don't ask a linear kernel to draw a line through them — you'll get ~50% accuracy. Look at a scatter of your data first. If a straight line would work, use Linear. Otherwise start with RBF — it handles almost anything.

Section 05

γ On RBF — The Reach Of Each Point's Influence

The gamma (γ) parameter in RBF controls how far each support vector's influence spreads. Think of each point holding a torch — high γ is a narrow beam, low γ is a floodlight.

🔦

Default To γ = 'scale'

sklearn's gamma='scale' auto-adapts to feature variance — it's the right starting point on almost every dataset. Only override with a specific value when you're actively grid-searching. Never leave γ at the legacy 'auto'.

Section 06

C — Wide Margin Or Tight Fit?

C range	Margin	Training accuracy	Risk
0.001 – 0.1	Very wide	Low	Underfits
0.5 – 5	Balanced	Medium	Good default
10 – 100	Narrow	High	Watch overfitting
1000+	Razor-thin	Very high	Severe overfitting

Section 07

Tune C And γ Together — The GridSearch Heatmap

The two knobs interact. High-C-with-high-γ is a disaster (memorises everything). Low-C-with-low-γ is another (underfits). Sweep both on a log grid and let CV find the sweet spot.

🎚️

Random Search Often Beats Grid Search

When you have three or more hyperparameters, RandomizedSearchCV with loguniform distributions finds better regions faster than a full grid. For plain (C, γ) tuning, either works — but never use linear spacing (range(1, 100)). Always log spacing: np.logspace(-3, 3, 7).

Section 08

Which Kernel Should I Use?

🎯

When In Doubt — RBF, Always

RBF is the safest first choice for anything that isn't obviously text or graph-shaped. Fit an RBF SVM with default parameters, look at the CV score. If it's decent, tune C and γ. If it's poor, try Linear (for high-dim) or Polynomial (if interactions might matter).

Section 09

Feature Scaling — Non-Negotiable For Non-Linear Kernels

📏 THE DISTANCE PROBLEM

RBF depends on ‖a − b‖² — and that depends on your units

RBF's similarity between two points is exp(−γ · ‖a − b‖²) — a function of Euclidean distance. Now imagine two features: income in rupees (values in millions) and age in years (values in tens). The distance calculation is completely dominated by income — the age feature might as well not exist.

StandardScaler puts every feature on the same footing (mean 0, std 1) so γ can act fairly on all of them. Skip scaling, and γ becomes meaningless.

💧

Always Scale Inside A Pipeline

Wrap StandardScaler and SVC inside a Pipeline so that cross-validation refits the scaler on each fold's training partition. Fit the scaler on train + test combined, and you leak test statistics into training — inflating every CV score by 2–5%.

Section 10

Implementation — Sklearn, Pipeline & Grid Search

from sklearn.svm import SVC, LinearSVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import loguniform
import numpy as np

# ── RBF default · always inside a pipeline ─────────────
rbf = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)),
])
rbf.fit(X_train, y_train)

# ── GridSearchCV · log-spaced C and γ ─────────────────
grid = GridSearchCV(rbf, param_grid={
    'clf__C':     np.logspace(-2, 2, 5),
    'clf__gamma': np.logspace(-4, 1, 6),
}, cv=5, scoring='roc_auc', n_jobs=-1).fit(X_train, y_train)
print(grid.best_params_)

# ── RandomizedSearchCV · better with ≥3 params ────────
rand = RandomizedSearchCV(rbf, param_distributions={
    'clf__C':     loguniform(1e-2, 1e2),
    'clf__gamma': loguniform(1e-4, 1e1),
}, n_iter=40, cv=5, scoring='roc_auc', n_jobs=-1).fit(X_train, y_train)

# ── Text · LinearSVC scales to millions of features ──
text = Pipeline([
    ('tfidf', TfidfVectorizer(ngram_range=(1, 2))),
    ('clf',   LinearSVC(C=1.0, max_iter=5000)),
]).fit(docs, labels)

# ── Polynomial · watch degree ────────────────────────
poly = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', SVC(kernel='poly', degree=3, coef0=1.0, C=1.0)),
])

⚡

Two Non-Negotiable Steps

Always wrap in a Pipeline so the scaler refits per CV fold. Always use loguniform distributions for C and γ in random search — linear ranges waste 90% of the budget in useless regions.

Section 11

Kernel Cheat Sheet — Formula, Params, Use Cases

Kernel	Formula	Key Params	Best For
Linear	aᵀ · b	C	Text · sparse · high-dim (LinearSVC scales to millions of features)
RBF (Gaussian)	exp(−γ‖a−b‖²)	C · γ	Default · non-linear boundaries of any shape
Polynomial	(γaᵀb + r)ᵈ	C · γ · degree · coef0	Feature interactions · vision · NLP
Sigmoid	tanh(γaᵀb + r)	C · γ · coef0	Rarely first choice · often use neural net instead
Precomputed	your K(a,b)	C	Custom similarity — graphs, strings, DNA, time series

🥇

Speed Winner

for large data

LinearSVC — O(n) training, handles millions of features. Kernel SVMs are O(n²) to O(n³).

🎯

Flexibility Winner

for unknown data

RBF — implicitly infinite-dimensional. Handles circles, spirals, XOR, almost anything.

🧩

Interpretability Winner

but only for linear

Linear — coefficients map directly to features, easy to explain. Non-linear kernels are black boxes.

Section 12

Common Pitfalls — What Silently Breaks SVM Kernels

📏

Unscaled Data + RBF

the #1 mistake

γ acts on Euclidean distance. Without StandardScaler, one large-scale feature dominates every kernel evaluation.

🎚️

Tuning C Alone

missing interactions

C and γ push against each other. Tune them jointly on log grids — never one at a time.

🔥

Huge γ On RBF

memorises noise

γ = 100+ wraps the boundary tightly around each training point. Support-vector count balloons; test accuracy collapses.

📉

Linear Search Ranges

wasted CPU

range(1, 100) for C wastes 99% of the budget in useless regions. Always use np.logspace.

🌀

Wrong Kernel Shape

50% accuracy

Linear kernel on concentric-circle data → random-guess accuracy. Plot the data first, or default to RBF.

🚫

Sigmoid As Default

unstable choice

Sigmoid isn't always a valid Mercer kernel. If it wins, that's a signal to try a neural network instead.

Section 13 · Part 1

Golden Rules — 1 to 3

✨ SVM KERNEL DISCIPLINE · RULES 1–3

1

Always scale features before non-linear kernels. Wrap StandardScaler and the SVM inside a Pipeline so cross-validation refits per fold. Skipping this destroys the geometry every non-linear kernel depends on.

2

Start with RBF, default gamma='scale'. It handles almost anything with sensible defaults. Only switch to Linear (for high-dim text) or Polynomial (for known feature interactions) when the data structure justifies it.

3

Tune C and γ jointly on log-spaced grids. Use np.logspace(-3, 3, 7) for both. They interact — tuning one at a time gives you the wrong optimum. Prefer RandomizedSearchCV with loguniform for larger searches.

Section 13 · Part 2

Golden Rules — 4 to 6

✨ SVM KERNEL DISCIPLINE · RULES 4–6

4

Monitor the train / validation gap. Train 0.99, CV 0.87 = overfit — usually from γ too large or C too large. Lower one, re-tune. Both curves converging low = underfit — reach for a more flexible kernel or lower C.

5

Watch the support-vector count. A healthy fit uses a modest fraction of training data as support vectors. When > 50% of points become support vectors, the model is memorising — reduce γ, or C, or both.

6

The kernel trick is free — dot products are all SVM needs. Never explicitly compute the high-dimensional lift. Let the kernel function do the work. For domain-specific similarity (graphs, strings, DNA), pass a precomputed kernel matrix.

✅

The Kernel Deployment Checklist

✅ Scaler in Pipeline · ✅ Started with RBF · gamma='scale' · ✅ C & γ tuned jointly on log grid · ✅ Train / CV gap acceptable · ✅ Support-vector count under 50% · ✅ probability=True set at construction (if needed).

FINAL

Kernels — The Magic That Curves A Linear Model

K(a,b)Similarity in a lifted space

4Kernels to know

RBFStart here · default

C × γTune jointly · log grid

📏Scale first · always

6Golden rules

🎯

The Foundation Is Set

You now understand what a kernel is, how the trick avoids computing the lift explicitly, how the four sklearn kernels differ in the boundaries they produce, how C and γ steer the bias-variance balance, and how to search for the right pair. Every kernel SVM you tune from now on rests on exactly these ideas.

📚

Where To Go Next

Study Kernel PCA (the same trick for dimensionality reduction) and Gaussian Processes (Bayesian cousin of RBF-SVM). For scale, learn Nyström approximation and SGDClassifier(loss='hinge'). Practise on Iris, MNIST, and Breast Cancer — RBF wins on all three with the right C and γ.

✨ End of tutorial · Press ← to review, or click Restart