Semantic Similarity & Sentence Embeddings

The Story That Explains Sentence Embeddings

Semantic similarity is the task of measuring how close two pieces of text are in meaning — not in exact words, not in length, but in the underlying idea they express. The engine that makes this possible is a sentence embedding: a dense numeric vector (typically 384 to 1536 numbers) that encodes the meaning of a sentence into a single point in high-dimensional space.

From Words to Vectors — A Brief History

Before we understand sentence embeddings, we need to understand why simple text matching fails, and how the field evolved to fix it.

What Is an Embedding, Mathematically?

An embedding is a function that maps a discrete object (a word, sentence, document) to a continuous vector in ℝⁿ. For sentence embeddings, a typical model produces a vector of 384 or 768 floating-point numbers for any input sentence, regardless of its length.

The Three Similarity Metrics

Once you have embeddings, you need a metric. The three most common choices are below. Cosine similarity is the default for NLP; the others have specific use cases.

Your First Sentence Embeddings — Step by Step

We'll use sentence-transformers, the most practical library for this task. It wraps SBERT models and handles all preprocessing automatically. Installation is one command.

# Install sentence-transformers (includes torch and transformers)
# pip install sentence-transformers

from sentence_transformers import SentenceTransformer, util
import numpy as np

# Load a lightweight, fast model (384-dimensional embeddings)
model = SentenceTransformer('all-MiniLM-L6-v2')

# Five sentences — notice they express similar ideas in very different words
sentences = [
    "The dog ran across the park at full speed.",
    "A canine sprinted through the garden quickly.",
    "The cat sat quietly by the window.",
    "She enjoyed her morning jog in the sunshine.",
    "Stock markets fell sharply on Tuesday.",
]

# Each sentence → 384-dimensional vector. Shape: (5, 384)
embeddings = model.encode(sentences, convert_to_tensor=True)
print(f"Embedding shape: {embeddings.shape}")

# Compute pairwise cosine similarity matrix
cosine_scores = util.cos_sim(embeddings, embeddings)

# Print the most similar pair (ignoring self-similarity)
for i in range(len(sentences)):
    for j in range(i+1, len(sentences)):
        print(f"Score: {cosine_scores[i][j]:.4f} | '{sentences[i][:40]}' ↔ '{sentences[j][:40]}'")

Story: The Paraphrase Problem

Popular Embedding Models — Which to Use?

Symmetric vs Asymmetric Semantic Similarity

Not all similarity tasks are the same. The type of task determines which model and approach to use. This is one of the most overlooked distinctions in practice.

Semantic Search — Full Working Example

Semantic search is the killer application of sentence embeddings. Pre-compute embeddings for your corpus once, store them, and at query time compute similarity between the query embedding and all stored embeddings. The top-k most similar become your search results.

from sentence_transformers import SentenceTransformer, util
import torch

model = SentenceTransformer('all-MiniLM-L6-v2')

# Corpus: articles on a medical FAQ site
corpus = [
    "Paracetamol reduces fever and mild to moderate pain. Take 500mg every 4–6 hours.",
    "Ibuprofen is a non-steroidal anti-inflammatory drug. Effective for headaches and inflammation.",
    "Staying hydrated helps the immune system fight infections and speeds recovery.",
    "Antibiotics treat bacterial infections but have no effect on viruses like the common cold.",
    "Regular exercise reduces cardiovascular disease risk and improves mental health.",
    "Diabetes type 2 can often be managed with diet, exercise, and weight loss alone.",
    "Vitamin D deficiency is linked to depression, fatigue, and weakened immunity.",
    "Sleep deprivation impairs cognitive function and weakens immune response.",
]

# Encode corpus ONCE and store (in production, save to disk or vector DB)
corpus_embeddings = model.encode(corpus, convert_to_tensor=True)

# User query — note completely different words from any corpus sentence
query = "What can I take for a headache and high temperature?"
query_embedding = model.encode(query, convert_to_tensor=True)

# Find top-3 most semantically similar corpus entries
hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=3)

print(f"Query: '{query}'\n")
print("Top 3 Results:")
for i, hit in enumerate(hits[0], 1):
    idx = hit['corpus_id']
    score = hit['score']
    print(f"  {i}. [Score: {score:.4f}] {corpus[idx]}")

Story: The Customer Support Revolution

Paraphrase Detection — Duplicate Question Finder

One of the cleanest use cases for semantic similarity is finding duplicate questions in a knowledge base or forum. The goal: two questions that ask the same thing in different words should be flagged as duplicates (think Stack Overflow, Quora, or internal wikis).

from sentence_transformers import SentenceTransformer, util
import itertools

model = SentenceTransformer('all-mpnet-base-v2')

questions = [
    "How do I reset my password?",
    "I forgot my login credentials, how can I recover access?",
    "What is the refund policy?",
    "Can I get my money back if I cancel?",
    "How do I change my billing information?",
    "Where can I update my credit card details?",
    "The app keeps crashing on startup.",
    "Application fails to launch every time I open it.",
]

embeddings = model.encode(questions, convert_to_tensor=True)

# Detect pairs above similarity threshold of 0.75
THRESHOLD = 0.75
pairs = util.paraphrase_mining(model, questions, threshold=THRESHOLD)

print(f"Duplicate question pairs (similarity > {THRESHOLD}):\n")
for score, i, j in pairs:
    print(f"  Score: {score:.4f}")
    print(f"    Q1: '{questions[i]}'")
    print(f"    Q2: '{questions[j]}'\n")

Clustering Sentences by Topic

When you have a large number of sentences (customer reviews, support tickets, survey responses), semantic clustering groups them automatically by topic — no labels required. This is unsupervised topic discovery at scale.

from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Customer reviews — mixed topics
reviews = [
    "Delivery was incredibly fast, arrived next day!",
    "Shipping took only 24 hours, very impressed.",
    "Package arrived damaged, very disappointed.",
    "Product was broken when I opened the box.",
    "Customer service team was very helpful and responsive.",
    "Support agent resolved my issue in minutes.",
    "The quality of the material feels cheap.",
    "Build quality is poor, fell apart after one week.",
]

embeddings = model.encode(reviews)

# Cluster into 4 groups (shipping / damage / support / quality)
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
labels = kmeans.fit_predict(embeddings)

print("Cluster Assignments:\n")
for cluster_id in range(4):
    print(f"  Cluster {cluster_id + 1}:")
    for i, label in enumerate(labels):
        if label == cluster_id:
            print(f"    - {reviews[i]}")
    print()

Scaling Up — Vector Databases

For small corpora (< 100,000 sentences), computing cosine similarity against all embeddings at query time is fast enough. For millions of entries, you need a vector database that enables approximate nearest-neighbour (ANN) search in milliseconds.

# FAISS — fast ANN search for large corpora
# pip install faiss-cpu   (or faiss-gpu for GPU)

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
DIM = 384  # embedding dimension of this model

# Step 1: Build the index (do this once, then save it)
index = faiss.IndexFlatIP(DIM)  # Inner Product = cosine sim for L2-normalised vectors

corpus = [
    "Python is a high-level general-purpose programming language.",
    "Machine learning enables computers to learn from data without explicit programming.",
    "Deep neural networks have revolutionised image recognition tasks.",
    "Natural language processing allows computers to understand human text.",
    "The Eiffel Tower is located in Paris, France.",
]

corpus_emb = model.encode(corpus, normalize_embeddings=True).astype(np.float32)
index.add(corpus_emb)  # Add vectors to index

# Step 2: Query at runtime (milliseconds even for millions of vectors)
query = "How do machines learn from examples?"
query_emb = model.encode([query], normalize_embeddings=True).astype(np.float32)

scores, indices = index.search(query_emb, k=3)  # top-3 results

print(f"Query: '{query}'\n")
for rank, (score, idx) in enumerate(zip(scores[0], indices[0]), 1):
    print(f"  {rank}. [{score:.4f}] {corpus[idx]}")

Cross-Encoder vs Bi-Encoder — The Accuracy Trade-off

All models we've used so far are bi-encoders: they embed sentences independently, then compare. Faster, but less accurate. A cross-encoder takes both sentences simultaneously for maximum accuracy — at the cost of speed.

from sentence_transformers import SentenceTransformer, CrossEncoder, util

# Stage 1: Bi-encoder for fast retrieval
bi_encoder = SentenceTransformer('multi-qa-MiniLM-L6-cos-v1')

# Stage 2: Cross-encoder for accurate reranking
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

corpus = [
    "Python can be used for machine learning and data science.",
    "Java is an object-oriented language primarily used in enterprise systems.",
    "scikit-learn provides tools for supervised and unsupervised learning in Python.",
    "PyTorch is a deep learning framework used for neural network training.",
    "TensorFlow was developed by Google and powers production AI systems worldwide.",
    "NumPy provides efficient array operations for numerical computing.",
]

query = "best Python tools for AI"

# Stage 1: Get top-3 candidates quickly
corpus_emb = bi_encoder.encode(corpus, convert_to_tensor=True)
query_emb  = bi_encoder.encode(query,  convert_to_tensor=True)
hits = util.semantic_search(query_emb, corpus_emb, top_k=3)[0]

candidates = [corpus[hit['corpus_id']] for hit in hits]

# Stage 2: Rerank with cross-encoder
pairs = [[query, c] for c in candidates]
rerank_scores = cross_encoder.predict(pairs)

ranked = sorted(zip(rerank_scores, candidates), reverse=True)
print("Reranked Results:")
for score, doc in ranked:
    print(f"  [{score:.4f}] {doc}")

Real-World Applications at a Glance

Golden Rules — Sentence Embeddings in Production