PCA Dimensional Reduction Numerical

The Story That Explains PCA

Principal Component Analysis (PCA) is an unsupervised linear transformation that projects data onto a new set of orthogonal axes (principal components), ordered by the amount of variance they explain. It is the workhorse of dimensionality reduction — used in data visualisation, noise filtering, feature extraction, and preprocessing before clustering or classification.

The Problem — Dataset & Goal

We have a small dataset with 4 samples and 2 features (x₁, x₂). The goal is to reduce it from 2D to 1D using PCA, retaining the maximum variance possible. We will solve every step by hand — exactly as you would in a Data Science exam notebook.

Step 1 — Compute the Mean of Each Feature

PCA is sensitive to the spread of data, not its absolute position. We must first shift the data so it is centred at the origin. Before that, we need the mean of each feature.

Step 2 — Mean-Centre the Data (Subtract the Mean)

We subtract the mean vector μ = [3, 4] from every data point to produce the centred matrix B. This shifts the data cloud so its centroid sits exactly at the origin — a prerequisite for covariance analysis.

Step 3 — Compute the Covariance Matrix

The covariance matrix C captures how much each pair of features varies together. It is a symmetric p × p matrix. For our 2-feature dataset it is 2 × 2.

Step 4 — Solve for Eigenvalues

The eigenvalues of the covariance matrix tell us how much variance each principal component explains. We find them by solving the characteristic equation: det(C − λI) = 0

Step 5 — Compute Eigenvectors (Principal Directions)

Eigenvectors give us the directions of the new axes. For each eigenvalue λ, we solve (C − λI)v = 0 and then normalise the resulting vector to unit length.

Step 6 — Explained Variance & Choosing Components

The eigenvalue tells us how much variance each principal component captures. We express this as a percentage of total variance to decide how many components to keep. This is the heart of the dimensionality reduction decision.

Step 7 — Project Data onto PC1 (Reduce to 1D)

We now project the mean-centred data matrix B onto the first principal component direction e₁. The projection is simply a dot product: multiply each centred data row by the eigenvector.

Full Solution Summary — One View

Here is the complete PCA solution in condensed form, exactly as you would write it in a final exam answer:

Key Formulae at a Glance

Three Concepts Every PCA Student Must Understand

When to Use PCA — and When Not To

Python Implementation — Verify Our Hand Calculation

Let us reproduce the exact numerical example above in Python — both manually and using scikit-learn. The outputs should match our hand-solved values to within floating-point precision.

import numpy as np
from sklearn.decomposition import PCA

# ── Original dataset ──────────────────────────────────────────
X = np.array([
    [1, 2],   # S1
    [3, 4],   # S2
    [5, 4],   # S3
    [3, 6],   # S4
], dtype=float)

# ── MANUAL PCA ────────────────────────────────────────────────
# Step 1: Mean vector
mu = X.mean(axis=0)
print(f"Mean vector μ: {mu}")   # [3. 4.]

# Step 2: Mean-centre
B = X - mu
print(f"\nCentred matrix B:\n{B}")

# Step 3: Covariance matrix (ddof=1 → divide by n-1)
C = np.cov(B, rowvar=False)     # equivalent to BᵀB / (n-1)
print(f"\nCovariance matrix C:\n{C}")

# Step 4 & 5: Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(C)

# eigh returns ascending order — reverse to descending
idx = np.argsort(eigenvalues)[::-1]
eigenvalues  = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

print(f"\nEigenvalues (sorted): {eigenvalues}")   # [4.  1.333]
print(f"Eigenvectors (columns):\n{eigenvectors}")

# Step 6: Explained variance
ev_ratio = eigenvalues / eigenvalues.sum()
print(f"\nExplained variance ratio: {ev_ratio}")  # [0.75  0.25]
print(f"Cumulative:              {ev_ratio.cumsum()}")

# Step 7: Project onto PC1 only
W  = eigenvectors[:, :1]      # take first column only
Z  = B @ W                      # matrix multiply
print(f"\nProjected data Z (1D):\n{Z.flatten()}")

# ── SCIKIT-LEARN PCA (one-liner verification) ─────────────────
pca = PCA(n_components=1)
Z_sk = pca.fit_transform(X)    # sklearn centres internally

print(f"sklearn Z:          {Z_sk.flatten()}")
print(f"sklearn EV ratio:   {pca.explained_variance_ratio_}")
print(f"sklearn eigenvalue: {pca.explained_variance_}")
print(f"sklearn component:  {pca.components_}")

PCA vs Other Dimensionality Reduction Methods

Golden Rules — PCA Non-Negotiables

The Real Significance of Eigenvectors

Most textbooks define eigenvectors as "directions of maximum variance" and move on. That is true but dangerously shallow. Here is what an eigenvector actually means — geometrically, algebraically, and in the context of your data.

The Real Significance of Eigenvalues

The eigenvalue is often summarised as "the variance explained." That is correct. But why is the eigenvalue the variance? Where does that come from? And what exactly does the number 4 mean in our example? Let us build the answer from scratch.

Eigenvalue and Eigenvector Together — The Complete Picture

They are inseparable. An eigenvalue without its eigenvector is just a number. An eigenvector without its eigenvalue is just a direction. Together they answer two questions that completely describe a principal component:

Practical Significance — What These Numbers Mean for Your Data