BERT Explained: Architecture, Masked Language Modelling

The Story That Explains BERT

BERT — Bidirectional Encoder Representations from Transformers — is a language model introduced by Google in 2018. Unlike earlier models that read text left-to-right (or right-to-left), BERT reads the entire sequence simultaneously, building a rich, context-aware representation for every token based on all surrounding tokens. It fundamentally changed the NLP landscape and forms the backbone of dozens of modern language understanding systems.

The Transformer Architecture — BERT's Foundation

BERT is built entirely on the Transformer Encoder — specifically a stack of encoder blocks from the original "Attention Is All You Need" paper (Vaswani et al., 2017). Understanding the encoder is essential to understanding BERT.

The Three Embedding Inputs

BERT Variants — Base vs Large

Self-Attention — How BERT "Sees" Everything at Once

Self-attention computes three vectors for every token — Query (Q), Key (K), and Value (V) — using learned weight matrices. The attention score between token i and token j is the dot product of token i's Query with token j's Key, scaled and then softmax-normalised.

Pre-Training Task 1 — Masked Language Model (MLM)

In MLM, 15% of input tokens are randomly selected for masking before the text is fed into the model. The model must predict the original identity of those tokens using the full surrounding context. Crucially, the masking follows a specific strategy to reduce the mismatch between pre-training (where [MASK] appears) and fine-tuning (where it does not).

MLM — Worked Example

Pre-Training Task 2 — Next Sentence Prediction (NSP)

NSP is a binary classification task. During pre-training, 50% of the time Sentence B is the true next sentence (IsNext), and 50% of the time it is a randomly sampled sentence from the corpus (NotNext). The final hidden state of the [CLS] token is passed through a linear layer + softmax to make this binary prediction.

Pre-Training Data — What BERT Learned From

BERT's Input Representation — The Full Picture

Every BERT input is constructed following a strict format. Understanding this format is essential when you implement fine-tuning, because you must replicate it exactly — the tokeniser and the model were built together.

Fine-Tuning — Adapting BERT to Your Task

Pre-training gives BERT a universal language model. Fine-tuning adapts that model to a specific downstream task by adding a small task-specific layer on top and training the whole stack end-to-end on labelled task data. Fine-tuning typically takes minutes to hours on a single GPU — orders of magnitude less compute than pre-training.

BERT in Code — From Tokenisation to Fine-Tuning

Step 1 — Install and Load

# Install Hugging Face Transformers
# pip install transformers torch datasets

from transformers import BertTokenizer, BertModel, BertForSequenceClassification
import torch

# Load pre-trained BERT-Base (uncased)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model     = BertModel.from_pretrained('bert-base-uncased')

Step 2 — Tokenise Text

text = 'The scientist discovered a new species in the rainforest.'

# Tokenise and encode to input IDs
tokens = tokenizer.tokenize(text)
print('Tokens:', tokens)
# Output: ['the', 'scientist', 'discovered', 'a', 'new', 'species',
#          'in', 'the', 'rain', '##forest', '.']

# Full encoding with attention mask and special tokens
encoding = tokenizer(
    text,
    return_tensors='pt',        # PyTorch tensors
    padding=True,
    truncation=True,
    max_length=128
)

print('Input IDs shape:', encoding['input_ids'].shape)
print('Token IDs:', encoding['input_ids'])

Step 3 — Extract Contextual Embeddings

model.eval()

with torch.no_grad():
    outputs = model(**encoding)

# last_hidden_state: shape [batch, seq_len, 768]
last_hidden = outputs.last_hidden_state
print('All token embeddings shape:', last_hidden.shape)

# [CLS] token embedding — position 0
cls_embedding = last_hidden[:, 0, :]
print('CLS embedding shape:', cls_embedding.shape)

# Pooled output (another classification-ready representation)
pooled = outputs.pooler_output
print('Pooled output shape:', pooled.shape)

Step 4 — Fine-Tune for Sentiment Classification

from transformers import BertForSequenceClassification, AdamW, get_scheduler
from torch.utils.data import DataLoader, Dataset
import torch

# ── Toy dataset ──────────────────────────────────────────────
class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.encodings = tokenizer(texts, truncation=True,
                                   padding=True, max_length=max_len,
                                   return_tensors='pt')
        self.labels = torch.tensor(labels)

    def __len__(self): return len(self.labels)

    def __getitem__(self, idx):
        return {key: val[idx] for key, val in self.encodings.items()}, self.labels[idx]

# Sample data
texts  = ['I loved this film!', 'Absolutely terrible.',
          'Best movie ever.',   'Waste of time.']
labels = [1, 0, 1, 0]   # 1 = positive, 0 = negative

dataset    = SentimentDataset(texts, labels, tokenizer)
dataloader = DataLoader(dataset, batch_size=2)

# ── Model — BERT-Base + classification head ───────────────────
clf_model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased',
    num_labels=2
)

optimizer = AdamW(clf_model.parameters(), lr=2e-5)

# ── Training loop (3 epochs) ─────────────────────────────────
clf_model.train()
for epoch in range(3):
    total_loss = 0
    for batch, batch_labels in dataloader:
        optimizer.zero_grad()
        outputs = clf_model(**batch, labels=batch_labels)
        loss     = outputs.loss
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1} | Loss: {total_loss/len(dataloader):.4f}")

Step 5 — Inference

clf_model.eval()

new_texts  = ['This is a masterpiece!', 'Completely boring.']
new_inputs = tokenizer(new_texts, return_tensors='pt',
                       padding=True, truncation=True)

with torch.no_grad():
    logits = clf_model(**new_inputs).logits

predictions = logits.argmax(dim=-1)
labels_map  = {0: 'Negative', 1: 'Positive'}

for text, pred in zip(new_texts, predictions):
    print(f"'{text}' → {labels_map[pred.item()]}")

MLM vs NSP — A Side-by-Side Comparison

BERT's Key Hyperparameters

BERT vs Other Models — Where It Stands

* SOP = Sentence Order Prediction — a harder version of NSP used in ALBERT.

BERT's Impact — Benchmark Results

When BERT was released, it achieved state-of-the-art on 11 NLP tasks simultaneously — an unprecedented leap. Here are representative results from the original paper:

Common Mistakes and How to Avoid Them

The BERT Pipeline — Complete Overview