Naive Bayes Classifier
Press Next → or use ← → arrow keys
What Is Naive Bayes?
Naive Bayes works exactly this way. It computes the probability of each class one feature at a time, multiplies them (assuming they're independent), scales by the class prior, and picks the class with the highest score.
Naive Bayes is a probabilistic classifier that applies Bayes' theorem with a strong assumption: every feature contributes independently to the probability of the class. Absurd in theory, effective in practice.
Bayes' Theorem — The Whole Model In One Line
When comparing classes for the same input, P(Features) is identical across every class — so it cancels. Naive Bayes only computes the numerator (likelihood × prior) for each class and picks the largest. This trick removes the hardest term from the math entirely.
The "Naive" Assumption — Features Are Independent
In real text, the words "free" and "money" almost always appear together — they are not independent. Naive Bayes assumes they are anyway. This assumption is usually wrong but it collapses an impossible computation into a trivial one, and it turns out that the classifier still picks the right class most of the time.
The Full Formula & Why We Live In Log-Space
A text of 500 words means 500 probabilities multiplied together. Each word probability might be around 0.001. Multiply five hundred of those and the result is effectively zero to a computer — underflow. The fix is universal: take the log of everything, add instead of multiply, and compare log-scores. sklearn handles this automatically.
Worked Example — Spam Or Ham?
Training: 10 emails · 5 spam · 5 ham. New email contains "free" and "winner", no "meeting".
| Feature | P( · | Spam ) | P( · | Ham ) |
|---|---|---|
| Prior — P(class) | 0.50 | 0.50 |
| "free" present | 0.80 | 0.20 |
| "winner" present | 0.60 | 0.10 |
| "meeting" absent | 1.00 | 0.40 |
Multiply the prior by all three feature likelihoods for each class. Spam wins 0.240 to 0.004. The email is classified as spam with very high confidence. That's Naive Bayes in one calculation — no gradient descent, no iterations, no matrix inversions.
Three Variants — Match The Distribution To Your Data
Using Multinomial NB on continuous data, or Gaussian NB on word counts — the model still runs, but silently gives worse answers. Continuous → Gaussian. Word counts → Multinomial. Binary flags → Bernoulli. This is the single most common Naive Bayes mistake.
Gaussian NB — Two Bell Curves, One Decision
For each continuous feature, Gaussian NB fits a bell curve to each class. To classify a new point, it reads the height of every class's bell at the observed value, multiplies by the priors, and picks the tallest score. Elegant, fast, exact — as long as the features really are bell-shaped.
Laplace Smoothing — The Zero-Probability Rescue
What if a word never appeared in a class during training? Its likelihood is exactly zero, and one zero in the product wipes the whole class out — regardless of every other feature.
α=1.0 is the safe default and almost always enabled in libraries. Tune α on a log grid
(0.01 – 2.0) via cross-validation for the last bit of performance. Setting α=0 silently
reintroduces the zero-frequency trap.
Advantages — Why Naive Bayes Still Ships
Disadvantages — Where It Falls Short
Where Naive Bayes Wins In Production
partial_fit. Ideal when data arrives continuously and models must stay fresh.For any text classification project, start with Multinomial Naive Bayes on TF-IDF. It takes ten lines of code, trains in seconds, and sets the accuracy floor every deeper model must clear before being worth the complexity.
Implementation — Sklearn In A Handful Of Lines
from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.pipeline import Pipeline from sklearn.calibration import CalibratedClassifierCV # ── Continuous features (glucose, BMI, age) ───── gnb = GaussianNB() gnb.fit(X_train, y_train) # ── Text classification — the classic combo ───── spam_pipe = Pipeline([ ('tfidf', TfidfVectorizer(ngram_range=(1, 2), min_df=2)), ('nb', MultinomialNB(alpha=1.0)), # Laplace smoothing on ]) spam_pipe.fit(emails_train, labels_train) print(spam_pipe.predict_proba(["Free winner prize!!"])) # ── Binary presence/absence flags ─────────────── bnb = BernoulliNB(alpha=1.0).fit(X_train_binary, y_train) # ── Fix over-confident probabilities ──────────── calibrated = CalibratedClassifierCV(spam_pipe, method='isotonic', cv=5) calibrated.fit(emails_train, labels_train) # ── Streaming / online learning ───────────────── mnb = MultinomialNB() for batch_X, batch_y in stream: mnb.partial_fit(batch_X, batch_y, classes=[0, 1])
Wrap the vectoriser and classifier in a Pipeline so cross-validation refits both per fold (no leakage). Wrap in CalibratedClassifierCV when you need honest probability estimates for decisions or thresholds. Skipping either leaves accuracy or trust on the table.
Common Pitfalls — What Silently Sinks Naive Bayes
α ≥ 0.01. Default 1.0 is safe.CalibratedClassifierCV before using for risk decisions.[0.01, 0.1, 1, 2] via CV — often a 1–3% boost.Golden Rules — Six Habits For Naive Bayes
CalibratedClassifierCV when the number
matters — thresholds, cost-sensitive decisions, ranking.
Naive Bayes — Simple, Fast, Surprisingly Sharp
You now understand Bayes' theorem, the naive independence assumption, the three sklearn variants, log-space arithmetic, and Laplace smoothing. Naive Bayes is the tiny, embarrassingly-fast classifier that still wins on text — and the baseline every other model has to beat before deployment.
Study Logistic Regression next — same probabilistic framing, no independence assumption. Then Linear SVM for max-margin text classification, and Calibration curves to fix NB's probability quirk. Practise on Kaggle's Spam Detection or IMDB Sentiment datasets.
🕵️ End of tutorial · Press ← to review, or click Restart