Word Embeddings

The Story That Explains Word Embeddings

Computers cannot read text. They can only process numbers. The naive solution — assigning each word an arbitrary integer (cat = 1, dog = 2, love = 3) — tells the model nothing about the relationship between words. Word Embeddings solve this by representing every word as a dense vector of real numbers — a point in high-dimensional space — where the position encodes semantic and syntactic meaning.

Why Not Just One-Hot Encode Words?

Before embeddings existed, NLP used one-hot encoding: a vector of all zeros except a single 1 at the word's index. With a vocabulary of 50,000 words, every word is a 50,000-dimensional vector. This works but fails catastrophically at scale.

Word2Vec — Teaching a Neural Network to Predict Context

Proposed by Tomas Mikolov at Google in 2013, Word2Vec is a two-layer neural network trained on a text corpus. It comes in two architectures: CBOW (Continuous Bag of Words) and Skip-gram. Both produce the same type of output — dense word vectors — but via opposite prediction tasks.

CBOW — Continuous Bag of Words

How CBOW Processes a Sentence — Step by Step

Skip-gram — The Reverse Problem

Skip-gram treats each (centre, context_word) as a separate training pair. Given window=2 and the word "brown", it generates four training examples: (brown, The), (brown, quick), (brown, fox), (brown, jumps). This makes Skip-gram better at learning representations for rare words because each occurrence produces more training signal.

Word Arithmetic — The Magic of Embedding Space

Word2Vec in Python — Full Working Example

from gensim.models import Word2Vec
import numpy as np

# ── 1. Tokenised corpus (each sentence is a list of words) ──
sentences = [
    ["the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"],
    ["she", "is", "a", "queen", "and", "he", "is", "a", "king"],
    ["paris", "is", "the", "capital", "of", "france"],
    ["berlin", "is", "the", "capital", "of", "germany"],
    ["machine", "learning", "is", "a", "subset", "of", "artificial", "intelligence"],
]

# ── 2. Train CBOW model ──────────────────────────────────────
cbow_model = Word2Vec(
    sentences    = sentences,
    vector_size  = 100,      # embedding dimension
    window       = 5,        # context window size
    min_count    = 1,        # include words seen at least once
    sg           = 0,        # 0 = CBOW, 1 = Skip-gram
    epochs       = 100,
    seed         = 42
)

# ── 3. Train Skip-gram model ─────────────────────────────────
sg_model = Word2Vec(
    sentences    = sentences,
    vector_size  = 100,
    window       = 5,
    min_count    = 1,
    sg           = 1,        # 1 = Skip-gram
    negative     = 5,        # negative samples
    epochs       = 100,
    seed         = 42
)

# ── 4. Access word vectors ───────────────────────────────────
king_vec = cbow_model.wv["king"]        # shape: (100,)
print(f"Vector shape: {king_vec.shape}")

# ── 5. Find most similar words ───────────────────────────────
similar = cbow_model.wv.most_similar("king", topn=3)
for word, score in similar:
    print(f"  {word:12s}: {score:.4f}")

# ── 6. Vector arithmetic (king - man + woman ≈ queen) ────────
result = cbow_model.wv.most_similar(
    positive=["king", "she"],
    negative=["he"],
    topn=1
)
print(f"king - he + she ≈ {result[0][0]}")

# ── 7. Cosine similarity ─────────────────────────────────────
sim = cbow_model.wv.similarity("king", "queen")
print(f"Similarity(king, queen) = {sim:.4f}")

GloVe — Global Vectors for Word Representation

Published by Pennington, Socher, and Manning at Stanford in 2014, GloVe (Global Vectors) combines the best of two worlds: the semantic richness of matrix factorisation methods and the shallow-network efficiency of Word2Vec.

The GloVe Objective Function

Using Pre-trained GloVe in Python

import numpy as np

# ── Load GloVe vectors from text file ───────────────────────
# Download: https://nlp.stanford.edu/projects/glove/
# glove.6B.100d.txt — 6B tokens, 400K vocab, 100-dim

def load_glove(path):
    embeddings = {}
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            parts = line.split()
            word   = parts[0]
            vector = np.array(parts[1:], dtype="float32")
            embeddings[word] = vector
    return embeddings

glove = load_glove("glove.6B.100d.txt")
print(f"Vocabulary size : {len(glove)}")        # 400,000
print(f"Vector dimension: {glove['king'].shape}")  # (100,)

# ── Cosine similarity helper ─────────────────────────────────
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print(cosine_sim(glove["king"], glove["queen"]))   # ~0.72
print(cosine_sim(glove["king"], glove["apple"]))   # ~0.10

# ── Analogy: king - man + woman ─────────────────────────────
target = glove["king"] - glove["man"] + glove["woman"]

best_word  = None
best_score = -1
for word, vec in glove.items():
    if word in {"king", "man", "woman"}:
        continue
    score = cosine_sim(target, vec)
    if score > best_score:
        best_score = score
        best_word  = word

print(f"king - man + woman ≈ {best_word}")   # queen

FastText — Embeddings for Every Word (Even Unseen Ones)

Developed by Facebook AI Research and published in 2016, FastText extends the Skip-gram model by representing each word as a sum of its character n-gram embeddings. This gives FastText three critical advantages over Word2Vec and GloVe.

FastText in Python

from gensim.models import FastText
import numpy as np

# ── 1. Train FastText model ──────────────────────────────────
sentences = [
    ["the", "quick", "brown", "fox", "jumps"],
    ["she", "is", "playing", "tennis"],
    ["the", "player", "played", "brilliantly"],
    ["machine", "learning", "powers", "modern", "nlp"],
]

model = FastText(
    sentences   = sentences,
    vector_size = 100,
    window      = 5,
    min_count   = 1,
    sg          = 1,     # Skip-gram (recommended for FastText)
    min_n       = 3,     # minimum n-gram length
    max_n       = 6,     # maximum n-gram length
    epochs      = 100,
    seed        = 42
)

# ── 2. OOV magic — unseen word gets a vector ─────────────────
# "playfulness" was NEVER in training data
oov_vec = model.wv["playfulness"]
print(f"OOV vector shape: {oov_vec.shape}")   # (100,) — not zero!

# ── 3. Morphological similarity ──────────────────────────────
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

sim_play_playing = cosine(model.wv["play"], model.wv["playing"])
sim_play_player  = cosine(model.wv["play"], model.wv["player"])
sim_play_fox     = cosine(model.wv["play"], model.wv["fox"])

print(f"play ↔ playing : {sim_play_playing:.4f}")   # high (shared n-grams)
print(f"play ↔ player  : {sim_play_player:.4f}")    # high
print(f"play ↔ fox     : {sim_play_fox:.4f}")       # low (no shared n-grams)

# ── 4. Use Facebook's pre-trained model ─────────────────────
# from gensim.models import FastText
# model = FastText.load_fasttext_format('cc.en.300.bin')
# Pre-trained on Common Crawl — 300 dims, 2M vocab

Word2Vec vs GloVe vs FastText — Complete Comparison

How Do You Know If Your Embeddings Are Any Good?

Embedding quality is evaluated on two categories of tasks: intrinsic (does the embedding space make linguistic sense?) and extrinsic (does using this embedding improve downstream task performance?).

End-to-End NLP Pipeline Using Pre-trained Embeddings

import numpy as np
from gensim.models import KeyedVectors

# ── Load Google News pre-trained Word2Vec ────────────────────
# Download: https://code.google.com/archive/p/word2vec/
wv = KeyedVectors.load_word2vec_format(
    "GoogleNews-vectors-negative300.bin",
    binary=True
)
print(f"Vocab: {len(wv)} words | Dim: {wv.vector_size}")

# ── Sentence embedding by averaging word vectors ─────────────
def sentence_vector(text, model, dim=300):
    tokens = text.lower().split()
    vecs   = [model[w] for w in tokens if w in model]
    if not vecs:
        return np.zeros(dim)
    return np.mean(vecs, axis=0)

# ── Simple sentiment classifier using embeddings ─────────────
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

reviews = [
    ("the movie was absolutely fantastic and thrilling", 1),
    ("terrible film waste of time boring plot",          0),
    ("brilliant acting and stunning cinematography",       1),
    ("awful predictable story bad acting",                0),
    ("outstanding performances moving story",             1),
    ("dull and painfully slow nothing happens",           0),
]

X = np.array([sentence_vector(r, wv) for r, _ in reviews])
y = np.array([label for _, label in reviews])

clf = LogisticRegression()
clf.fit(X, y)

test_sentences = [
    "wonderful magical experience loved every moment",
    "completely unwatchable dreadful acting"
]

for s in test_sentences:
    vec   = sentence_vector(s, wv).reshape(1, -1)
    pred  = clf.predict(vec)[0]
    label = "POSITIVE" if pred == 1 else "NEGATIVE"
    print(f"{label}: {s}")

Golden Rules for Word Embeddings

Pre-trained Models & Resources