Backpropagation Algorithm

The Story That Makes Backpropagation Click

Backpropagation (short for "backward propagation of errors") is the algorithm that makes neural network learning possible. It computes how much each weight in the network contributed to the final loss, using the chain rule of calculus applied efficiently through the computation graph — all weights updated in a single backward sweep.

The Four Core Concepts

The Computation Graph — Your Network's Blueprint

Every neural network operation can be drawn as a graph of primitive computations. During the forward pass, data flows left to right. During the backward pass, gradients flow right to left. The graph makes it possible to apply the chain rule mechanically, node by node.

The Chain Rule — The Mathematical Engine

The chain rule states: if L depends on z through a, then:

Full Numerical Example — Step-by-Step Mathematics

We will build a tiny neural network with 2 inputs → 1 hidden neuron → 1 output neuron and walk through every single number in both the forward and backward pass. No hand-waving, no skipping — every calculation shown.

👉 Step 1 — Forward Pass

👉 Step 2 — Backward Pass

Now we work backwards from the loss, computing gradients layer by layer using the chain rule. We start at the output and propagate the error signal upstream.

👉 Step 3 — Weight Update (Gradient Descent)

Layer-by-Layer Diagram — What Each Layer Computes

Complete Gradient Summary

Python Implementation — From Scratch

Below is a clean, well-commented Python implementation of the exact network we computed by hand above. It verifies all our numbers programmatically.

import numpy as np

# ── Network weights (matching our manual example) ──────────
W1 = np.array([[0.4, 0.6]])   # shape (1, 2) — 1 hidden neuron, 2 inputs
b1 = np.array([[0.1]])         # shape (1, 1)
W2 = np.array([[0.9]])         # shape (1, 1) — 1 output, 1 hidden neuron
b2 = np.array([[0.1]])         # shape (1, 1)

x  = np.array([[0.5], [0.8]])  # shape (2, 1) — column vector
y  = np.array([[1.0]])         # true label

# ── Activation functions ───────────────────────────────────
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_deriv(z):
    s = sigmoid(z)
    return s * (1 - s)          # σ'(z) = σ(z)(1 - σ(z))

# ── FORWARD PASS ──────────────────────────────────────────
z1 = W1 @ x + b1              # pre-activation hidden:  (1,1)
a1 = sigmoid(z1)              # activation hidden

z2 = W2 @ a1 + b2             # pre-activation output: (1,1)
y_hat = sigmoid(z2)           # prediction ŷ

loss = 0.5 * (y_hat - y)**2   # MSE loss

print(f"z1    = {z1[0,0]:.4f}")
print(f"a1    = {a1[0,0]:.4f}")
print(f"z2    = {z2[0,0]:.4f}")
print(f"y_hat = {y_hat[0,0]:.4f}")
print(f"Loss  = {loss[0,0]:.4f}")

# ── BACKWARD PASS ─────────────────────────────────────────
# Output layer
dL_dyhat = y_hat - y                    # dL/dŷ
delta2   = dL_dyhat * sigmoid_deriv(z2) # δ₂ = dL/dz₂
dL_dW2   = delta2 @ a1.T               # dL/dW₂
dL_db2   = delta2                       # dL/db₂

# Hidden layer
dL_da1 = W2.T @ delta2                 # propagate error upstream
delta1 = dL_da1 * sigmoid_deriv(z1)   # δ₁ = dL/dz₁
dL_dW1 = delta1 @ x.T                  # dL/dW₁  shape (1,2)
dL_db1 = delta1                         # dL/db₁

print(f"\nGradients:")
print(f"delta2  = {delta2[0,0]:.4f}  (output error signal)")
print(f"dL/dW2  = {dL_dW2[0,0]:.4f}")
print(f"dL/db2  = {dL_db2[0,0]:.4f}")
print(f"delta1  = {delta1[0,0]:.4f}  (hidden error signal)")
print(f"dL/dW11 = {dL_dW1[0,0]:.4f}")
print(f"dL/dW12 = {dL_dW1[0,1]:.4f}")
print(f"dL/db1  = {dL_db1[0,0]:.4f}")

# ── GRADIENT DESCENT UPDATE ───────────────────────────────
lr = 0.5  # learning rate η

W2 = W2 - lr * dL_dW2
b2 = b2 - lr * dL_db2
W1 = W1 - lr * dL_dW1
b1 = b1 - lr * dL_db1

print(f"\nUpdated Weights:")
print(f"W2  = {W2[0,0]:.4f}")
print(f"b2  = {b2[0,0]:.4f}")
print(f"W11 = {W1[0,0]:.4f}  W12 = {W1[0,1]:.4f}")
print(f"b1  = {b1[0,0]:.4f}")

The Full Training Loop — 1000 Iterations

import numpy as np
import matplotlib.pyplot as plt

# ── Data: XOR problem ─────────────────────────────────────
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float).T  # (2,4)
Y = np.array([[0,1,1,0]], dtype=float)              # (1,4)

# ── Initialise weights with small random values ───────────
np.random.seed(42)
W1 = np.random.randn(4, 2) * 0.5   # 4 hidden neurons
b1 = np.zeros((4, 1))
W2 = np.random.randn(1, 4) * 0.5
b2 = np.zeros((1, 1))

lr     = 2.0
losses = []

for epoch in range(5000):
    # ── Forward ────────────────────────────────────────────
    z1    = W1 @ X + b1             # (4,4)
    a1    = 1 / (1 + np.exp(-z1))  # sigmoid
    z2    = W2 @ a1 + b2            # (1,4)
    y_hat = 1 / (1 + np.exp(-z2))

    loss  = np.mean(0.5 * (y_hat - Y)**2)
    losses.append(loss)

    # ── Backward ───────────────────────────────────────────
    m = X.shape[1]                   # number of samples = 4

    delta2 = (y_hat - Y) * y_hat * (1 - y_hat) / m
    dW2    = delta2 @ a1.T
    db2    = np.sum(delta2, axis=1, keepdims=True)

    dA1    = W2.T @ delta2
    delta1 = dA1 * a1 * (1 - a1)
    dW1    = delta1 @ X.T
    db1    = np.sum(delta1, axis=1, keepdims=True)

    # ── Update ─────────────────────────────────────────────
    W2 -= lr * dW2;  b2 -= lr * db2
    W1 -= lr * dW1;  b1 -= lr * db1

    if epoch % 1000 == 0:
        print(f"Epoch {epoch:5d}  Loss: {loss:.5f}")

# ── Final predictions ─────────────────────────────────────
print(f"\nFinal predictions:")
print(np.round(y_hat, 2), "  (target: [0,1,1,0])")

The network learned the XOR function — a classic non-linear problem — purely through repeated forward + backward passes. No magic: just the chain rule, applied thousands of times.

Common Pitfalls in Backpropagation

Golden Rules of Backpropagation

Activation Functions & Their Derivatives

PyTorch — Automatic Backpropagation

In practice, frameworks like PyTorch compute all gradients automatically. But understanding the manual derivation is essential — it tells you why the API works the way it does.

import torch
import torch.nn as nn

# ── Reproduce our manual example in PyTorch ───────────────
x     = torch.tensor([[0.5], [0.8]], dtype=torch.float32)
y     = torch.tensor([[1.0]])

# Weights — require_grad=True tracks the computation graph
W1 = torch.tensor([[0.4, 0.6]], requires_grad=True)
b1 = torch.tensor([[0.1]], requires_grad=True)
W2 = torch.tensor([[0.9]], requires_grad=True)
b2 = torch.tensor([[0.1]], requires_grad=True)

# ── Forward pass (PyTorch builds the computation graph) ───
z1    = W1 @ x + b1
a1    = torch.sigmoid(z1)
z2    = W2 @ a1 + b2
y_hat = torch.sigmoid(z2)
loss  = 0.5 * (y_hat - y)**2

# ── Backward pass (ONE LINE — PyTorch does all the chain rule)
loss.backward()

# ── Gradients are now in .grad attributes ─────────────────
print(f"dL/dW2  = {W2.grad.item():.4f}")
print(f"dL/db2  = {b2.grad.item():.4f}")
print(f"dL/dW11 = {W1.grad[0,0].item():.4f}")
print(f"dL/dW12 = {W1.grad[0,1].item():.4f}")
print(f"dL/db1  = {b1.grad.item():.4f}")

# ── Gradient descent update ───────────────────────────────
with torch.no_grad():
    W2 -= 0.5 * W2.grad
    b2 -= 0.5 * b2.grad
    W1 -= 0.5 * W1.grad
    b1 -= 0.5 * b1.grad

print(f"\nUpdated: W2={W2.item():.4f}  b2={b2.item():.4f}")
print(f"         W1={W1.detach().numpy()}  b1={b1.item():.4f}")