Content-Based Filtering in Recommendation

Content-Based Filtering — The Idea Behind "More Like This"

Content-based filtering is one of the two pillars of recommender systems (the other being collaborative filtering). It works entirely from item descriptions — no other users are needed. This makes it ideal when you have rich text data about your items, a cold-start problem with new users, or privacy concerns that prevent sharing user data.

The Complete Pipeline — Bird's Eye View

Before diving into the maths, here is the end-to-end flow of our movie recommendation system. Each step is a section in this tutorial.

Feature Engineering — Turning Metadata Into Text Soup

What Features Can We Use for Movies?

Building the Feature Soup

The standard approach is to concatenate all relevant features into a single string per movie, then apply text preprocessing. Here is what that looks like in practice:

Text Preprocessing Steps

import pandas as pd
import numpy as np
from ast import literal_eval

# ── Load the TMDB dataset ──────────────────────────────────
df = pd.read_csv('tmdb_5000_movies.csv')
credits = pd.read_csv('tmdb_5000_credits.csv')

# Merge on movie title
df = df.merge(credits, left_on='title', right_on='title')

# ── Parse JSON columns ─────────────────────────────────────
json_cols = ['genres', 'keywords', 'cast', 'crew']
for col in json_cols:
    df[col] = df[col].apply(literal_eval)

# ── Extract top-3 cast members ─────────────────────────────
def get_top_cast(cast_list, n=3):
    return [member['name'].replace(' ', '').lower()
            for member in cast_list[:n]]

# ── Extract director from crew ─────────────────────────────
def get_director(crew_list):
    for person in crew_list:
        if person['job'] == 'Director':
            return [person['name'].replace(' ', '').lower()] * 2  # weight director x2
    return []

# ── Generic list cleaner ───────────────────────────────────
def clean_list(lst):
    return [item['name'].replace(' ', '').lower() for item in lst]

# ── Apply feature extraction ───────────────────────────────
df['cast_clean']     = df['cast'].apply(get_top_cast)
df['director_clean'] = df['crew'].apply(get_director)
df['genres_clean']   = df['genres'].apply(clean_list)
df['keywords_clean'] = df['keywords'].apply(clean_list)

# ── Combine into the feature soup ─────────────────────────
def create_soup(row):
    return ' '.join(
        row['keywords_clean'] +
        row['genres_clean']   * 2 +   # genres weighted x2
        row['cast_clean']    +
        row['director_clean']        # director already x2
    )

df['soup'] = df.apply(create_soup, axis=1)

print(df['soup'].iloc[0])

TF-IDF — Measuring Word Importance Mathematically

The Two Components

Worked Example — 5 Movies, 5 Words

Notice: the word the scores 0 everywhere — it appears in every movie and has IDF ≈ 0. The word dream scores high only for Inception, because it is rare across all other movies. This is TF-IDF at work.

Implementing TF-IDF in Scikit-Learn

from sklearn.feature_extraction.text import TfidfVectorizer

# ── Fit TF-IDF on the feature soup ──────────────────────────
tfidf = TfidfVectorizer(
    stop_words='english',     # auto-remove common English words
    min_df=2,                  # ignore words appearing in < 2 movies
    max_features=10000,        # cap vocabulary at 10k most frequent
    ngram_range=(1, 2)          # include bigrams like "space exploration"
)

# Fill any NaN overviews
df['soup'] = df['soup'].fillna('')

# tfidf_matrix shape: (num_movies, num_unique_tokens)
tfidf_matrix = tfidf.fit_transform(df['soup'])

print(f"Matrix shape: {tfidf_matrix.shape}")
print(f"Total unique tokens: {len(tfidf.vocabulary_)}")

# Peek at top tokens for Interstellar
movie_idx = df[df['title'] == 'Interstellar'].index[0]
feature_names = tfidf.get_feature_names_out()
scores = tfidf_matrix[movie_idx].toarray()[0]
top10 = sorted(zip(feature_names, scores),
               key=lambda x: -x[1])[:10]

for token, score in top10:
    print(f"  {token:20s}: {score:.4f}")

Cosine Similarity — Measuring the Angle Between Movies

The Formula

Visual Intuition — 2D Example

Computing the Full Similarity Matrix

We compute pairwise cosine similarity for all movies at once. For 5,000 movies this produces a 5,000 × 5,000 matrix — 25 million values. Each cell [i][j] holds the similarity between movie i and movie j.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# ── Compute full pairwise cosine similarity ──────────────────
# tfidf_matrix is (4803, 8742) — sparse
cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)
print(f"Similarity matrix shape: {cosine_sim.shape}")

# ── Build a reverse-lookup index: title → row index ─────────
indices = pd.Series(df.index, index=df['title']).drop_duplicates()

# ── Peek at similarity for Interstellar ─────────────────────
idx = indices['Interstellar']
sim_scores = list(enumerate(cosine_sim[idx]))
sim_scores = sorted(sim_scores, key=lambda x: -x[1])

print("\nTop 5 similar to Interstellar:")
for movie_idx, score in sim_scores[1:6]:   # skip [0] — it's itself
    title = df['title'].iloc[movie_idx]
    print(f"  {title:30s}: {score:.4f}")

Item Similarity — Understanding the Similarity Matrix

The cosine similarity matrix is the heart of our recommendation engine. Let's visualise what it looks like and how to interpret it.

The Complete Movie Recommendation System — Full Project

Let's bring all five concepts together into a single, production-ready Python script. We will use the TMDB 5000 Movies dataset — 4,803 movies with full metadata.

# ═══════════════════════════════════════════════════════════
# Movie Recommendation System — Content-Based Filtering
# Dataset: TMDB 5000 Movies
# ═══════════════════════════════════════════════════════════

import pandas as pd
import numpy as np
from ast import literal_eval
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# ── 1. LOAD DATA ────────────────────────────────────────────
movies  = pd.read_csv('tmdb_5000_movies.csv')
credits = pd.read_csv('tmdb_5000_credits.csv')
df = movies.merge(credits, on='title')

# Keep only what we need
df = df[['title', 'overview', 'genres',
         'keywords', 'cast', 'crew']].copy()
df.dropna(subset=['overview'], inplace=True)

# ── 2. PARSE JSON COLUMNS ───────────────────────────────────
for col in ['genres', 'keywords', 'cast', 'crew']:
    df[col] = df[col].apply(literal_eval)

# ── 3. FEATURE EXTRACTION ───────────────────────────────────
def extract_names(lst, n=None):
    names = [x['name'].replace(' ', '').lower() for x in lst]
    return names[:n] if n else names

def get_director(crew):
    directors = [x['name'].replace(' ', '').lower()
                 for x in crew if x['job'] == 'Director']
    return directors * 2   # weight director twice

df['genres_f']   = df['genres'].apply(lambda x: extract_names(x) * 2)
df['keywords_f'] = df['keywords'].apply(lambda x: extract_names(x))
df['cast_f']     = df['cast'].apply(lambda x: extract_names(x, n=3))
df['crew_f']     = df['crew'].apply(get_director)

# ── 4. BUILD FEATURE SOUP ───────────────────────────────────
def build_soup(row):
    overview_words = str(row['overview']).lower().split()
    meta = (row['genres_f'] + row['keywords_f'] +
            row['cast_f']   + row['crew_f'])
    return ' '.join(overview_words + meta)

df['soup'] = df.apply(build_soup, axis=1)

# ── 5. TF-IDF VECTORISATION ─────────────────────────────────
tfidf = TfidfVectorizer(
    stop_words='english',
    min_df=2,
    max_features=15000,
    ngram_range=(1, 2),
    sublinear_tf=True    # dampen very high term freqs
)
tfidf_matrix = tfidf.fit_transform(df['soup'])

# ── 6. COSINE SIMILARITY MATRIX ─────────────────────────────
cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)

# Title-to-index lookup
indices = pd.Series(df.index, index=df['title']).drop_duplicates()

# ── 7. RECOMMENDATION FUNCTION ──────────────────────────────
def recommend(title, n=10):
    """Return top-n content-based recommendations for a given movie title."""
    if title not in indices:
        return f"Movie '{title}' not found in dataset."

    idx        = indices[title]
    sim_scores = list(enumerate(cosine_sim[idx]))
    sim_scores = sorted(sim_scores, key=lambda x: -x[1])
    sim_scores = sim_scores[1:n+1]  # skip self (index 0)

    movie_indices = [i[0] for i in sim_scores]
    scores        = [round(i[1], 4) for i in sim_scores]

    result = df['title'].iloc[movie_indices].reset_index(drop=True)
    return pd.DataFrame({'Recommended Movie': result, 'Similarity Score': scores})

# ── 8. TEST IT ──────────────────────────────────────────────
print("🎬 Recommendations for: Interstellar\n")
print(recommend('Interstellar', n=8).to_string(index=False))

Evaluating & Tuning Your Recommender

Content-based recommenders are harder to evaluate than classifiers — there is no single "correct" answer. Here are the standard approaches:

# ── Quick Precision@K evaluation with mock ratings ──────────
def precision_at_k(recommended_titles, liked_titles, k):
    top_k = recommended_titles[:k]
    hits  = len(set(top_k) & set(liked_titles))
    return hits / k

# Example: user liked these space films
user_liked = ['Gravity', 'The Martian', 'Arrival', 'Contact']

recs = recommend('Interstellar', n=10)['Recommended Movie'].tolist()
p5  = precision_at_k(recs, user_liked, k=5)
p10 = precision_at_k(recs, user_liked, k=10)
print(f"Precision@5  : {p5:.2f}")
print(f"Precision@10 : {p10:.2f}")

Golden Rules