Bag of Words, TF-IDF & N-grams

Why Machines Cannot Read Words — The Core Problem

In NLP (Natural Language Processing), before you build any classifier, sentiment analyser, or search engine, you must first answer one fundamental question: how do I turn this text into numbers? The three classical answers are Bag of Words (BoW), TF-IDF, and N-grams. Each represents a different philosophy, a different trade-off, and a different view of what "meaning" is in text.

Bag of Words (BoW) — The Simplest Possible Idea

How BoW Works — Step by Step

Concrete Example

Consider these three short documents:

Each document is now a 6-dimensional vector. D1 = [0, 0, 0, 1, 1, 1]. This is what gets fed to your logistic regression, SVM, or naive Bayes classifier.

BoW in Python — Full Working Example

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

# Our corpus
corpus = [
    "I love machine learning",
    "machine learning is great",
    "I love great food",
]

# Build the BoW model
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)

# Inspect the vocabulary
vocab = vectorizer.get_feature_names_out()
print("Vocabulary:", vocab)

# View as a readable DataFrame
df = pd.DataFrame(
    X.toarray(),
    columns=vocab,
    index=["D1", "D2", "D3"]
)
print(df)

BoW — Strengths and Weaknesses

TF-IDF — Rewarding Rare, Punishing Common

The Two Components of TF-IDF

TF-IDF Worked Calculation

TF-IDF Visual — How Scores Change Across Documents

TF-IDF in Python

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
import numpy as np

corpus = [
    "I love machine learning",
    "machine learning is great",
    "I love great food",
]

# Build TF-IDF model
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)

vocab = tfidf.get_feature_names_out()
df = pd.DataFrame(
    np.round(X.toarray(), 4),
    columns=vocab,
    index=["D1", "D2", "D3"]
)

print(df)

# Inspect IDF scores — higher = rarer word
for word, idf in zip(vocab, tfidf.idf_):
    print(f"  {word:12s}: IDF = {idf:.4f}")

BoW vs TF-IDF — Side by Side

N-grams — Capturing Context and Word Sequences

N-gram Types — Visual Breakdown

N-grams in Python

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

corpus = [
    "machine learning is not hard",
    "deep learning is great and not boring",
    "not all machine learning is deep learning",
]

# ── Unigrams only (standard BoW) ─────────────────────────────
uni = CountVectorizer(ngram_range=(1, 1))
X_uni = uni.fit_transform(corpus)
print("Unigram features:", uni.get_feature_names_out())

# ── Bigrams only ─────────────────────────────────────────────
bi = CountVectorizer(ngram_range=(2, 2))
X_bi = bi.fit_transform(corpus)
print("\nBigram features:", bi.get_feature_names_out())

# ── Unigrams + Bigrams combined ───────────────────────────────
combo = CountVectorizer(ngram_range=(1, 2))
X_combo = combo.fit_transform(corpus)
print("\nCombined (1-2) features:", len(combo.get_feature_names_out()),
      "total features")
print(combo.get_feature_names_out())

# ── TF-IDF with bigrams (common in practice) ──────────────────
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_bi = TfidfVectorizer(ngram_range=(1, 2), max_features=20)
X_tfidf_bi = tfidf_bi.fit_transform(corpus)
print("\nTF-IDF + Bigrams top features:")
print(tfidf_bi.get_feature_names_out())

Full Pipeline — Sentiment Analysis on Movie Reviews

Let's put all three techniques together on a real task: classifying movie reviews as positive or negative. We compare BoW, TF-IDF, and TF-IDF+Bigrams directly.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import numpy as np

# ── Load a binary subset: comp.graphics vs sci.space ─────────
cats = ['rec.sport.hockey', 'sci.space']
data = fetch_20newsgroups(subset='all', categories=cats,
                           remove=('headers', 'footers', 'quotes'))

X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)

# ── Method 1: Bag of Words ─────────────────────────────────
bow = CountVectorizer(stop_words='english', max_features=10000)
X_bow_tr = bow.fit_transform(X_train)
X_bow_te = bow.transform(X_test)
lr_bow = LogisticRegression(max_iter=500)
lr_bow.fit(X_bow_tr, y_train)
acc_bow = accuracy_score(y_test, lr_bow.predict(X_bow_te))

# ── Method 2: TF-IDF (unigrams) ────────────────────────────
tfidf = TfidfVectorizer(stop_words='english', max_features=10000)
X_tfidf_tr = tfidf.fit_transform(X_train)
X_tfidf_te = tfidf.transform(X_test)
lr_tfidf = LogisticRegression(max_iter=500)
lr_tfidf.fit(X_tfidf_tr, y_train)
acc_tfidf = accuracy_score(y_test, lr_tfidf.predict(X_tfidf_te))

# ── Method 3: TF-IDF + Bigrams ─────────────────────────────
tfidf_bi = TfidfVectorizer(ngram_range=(1, 2), stop_words='english', max_features=10000)
X_bi_tr = tfidf_bi.fit_transform(X_train)
X_bi_te = tfidf_bi.transform(X_test)
lr_bi = LogisticRegression(max_iter=500)
lr_bi.fit(X_bi_tr, y_train)
acc_bi = accuracy_score(y_test, lr_bi.predict(X_bi_te))

# ── Compare ────────────────────────────────────────────────
print(f"BoW Accuracy:         {acc_bow:.4f}")
print(f"TF-IDF Accuracy:      {acc_tfidf:.4f}")
print(f"TF-IDF+Bigrams Acc.:  {acc_bi:.4f}")

# ── Top TF-IDF features for each class ────────────────────
feature_names = tfidf_bi.get_feature_names_out()
for cls, name in enumerate(data.target_names):
    top = np.argsort(lr_bi.coef_[cls])[-10:]
    print(f"\nTop features for '{name}':")
    print([feature_names[i] for i in top])

Head-to-Head — When to Use Which Technique

Shared Limitations — What These Methods Cannot Do

Despite their wide use, all three classical methods share a fundamental blind spot: they represent text as independent word occurrences. They have no understanding of what words actually mean in context.

Golden Rules — Non-Negotiable Practices