Principal Component Analysis (PCA)

The Story That Explains PCAवह कहानी जो PCA समझाती है一个解释 PCA 的故事

Principal Component Analysis (PCA) is an unsupervised, linear method. It does not look at labels. It describes the same data in a new set of axes, called principal components, sorted by how much variance each one explains. PC1 is the direction of the largest spread. PC2 is the direction of the largest remaining spread that is at 90° to PC1, and so on.

Principal Component Analysis (PCA) एक unsupervised, linear तरीका है। यह labels नहीं देखता। यह उसी data को नए axes में लिखता है, जिन्हें principal components कहते हैं, और उन्हें इस क्रम में रखता है कि हर एक कितना variance समझाता है। PC1 सबसे बड़े फैलाव की दिशा है। PC2 बचे हुए सबसे बड़े फैलाव की दिशा है जो PC1 से 90° पर हो, और इसी तरह आगे।

主成分分析（PCA）是一种无监督的线性方法，它不看标签。它用一组新的坐标轴来描述同样的数据，这些轴叫作主成分，并按每个轴解释的方差大小排序。PC1 是散布最大的方向；PC2 是与 PC1 成 90° 的方向中散布最大的；依此类推。

PCA in Pictures — Understand It Before Any Mathsतस्वीरों में PCA — गणित से पहले समझें图解 PCA — 先理解，再学数学

Forget formulas for a moment. PCA does one simple job: it replaces many columns with a few new columns, and keeps as much information as possible. The four pictures below show how.

एक पल के लिए formulas भूल जाइए। PCA एक सरल काम करता है: यह बहुत सारे columns की जगह कुछ नए columns रखता है, और जितनी हो सके उतनी जानकारी बचाता है। नीचे की चार तस्वीरें दिखाती हैं कि कैसे।

先忘掉公式。PCA 只做一件简单的事：用少数几个新列代替很多列，同时尽量多地保留信息。下面四张图告诉你它是怎么做到的。

Picture 4 — The PCA Words, in Picturesतस्वीर 4 — PCA के शब्द, तस्वीरों में图 4 — 用图片解释 PCA 术语

Six words appear again and again in PCA. Here is what each one means in plain language.

PCA में छह शब्द बार-बार आते हैं। यहाँ हर एक का सरल मतलब है।

PCA 中有六个词反复出现。下面用大白话解释每一个。

The Same Idea in 2-D, With Numbers2-D में वही विचार, Numbers के साथ同样的想法：二维 + 数字

Where Is PCA Used in Real Life?असली दुनिया में PCA कहाँ इस्तेमाल होता है?现实生活中哪里用到 PCA？

Why Dimensionality Reduction MattersDimensionality Reduction क्यों ज़रूरी है为什么降维很重要

Modern datasets can have hundreds or thousands of features. More features sounds better, but past a point it causes three real problems:

आज के datasets में सैकड़ों या हज़ारों features हो सकते हैं। ज़्यादा features अच्छा लगता है, पर एक हद के बाद इससे तीन असली समस्याएँ आती हैं:

现代数据集可能有成百上千个特征。特征越多听起来越好，但超过一定程度就会带来三个实际问题：

The PCA Algorithm — Five StepsPCA Algorithm — पाँच StepsPCA 算法 — 五个步骤

The Maths — Made Concrete with Numbersगणित — Numbers के साथ数学 — 用具体数字说明

Numerical Example: 2 Features → 1 Principal ComponentNumerical Example: 2 Features → 1 Principal Component数值例子：2 个特征 → 1 个主成分

Let's run PCA by hand on a tiny dataset: 5 students with two exam scores. We want the one direction (PC1) that best summarises both scores. Both scores use the same 0–100 scale, so here we only centre the data and do not divide by the standard deviation.

चलिए एक छोटे dataset पर हाथ से PCA करते हैं: 5 students और दो exam scores. हमें वह एक दिशा (PC1) चाहिए जो दोनों scores को सबसे अच्छी तरह summarise करे। दोनों scores एक ही 0–100 scale पर हैं, इसलिए यहाँ हम data को सिर्फ़ centre करते हैं, standard deviation से भाग नहीं देते।

我们在一个很小的数据集上手算 PCA：5 个学生、两门考试成绩。我们想找到最能概括两门成绩的那一个方向（PC1）。两门成绩都是 0–100 分制，所以这里只做中心化，不除以标准差。

Key Formulasमुख्य Formulas关键公式

Visual Diagram — What PCA Does in 3-DVisual Diagram — 3-D में PCA क्या करता है可视化 — PCA 在三维中做了什么

Here is a real 3-D cloud of 90 points. The three arrows are its real principal components, computed in your browser from the covariance matrix. Arrow length grows with √λ.

यह 90 points का एक असली 3-D cloud है। तीनों तीर इसके असली principal components हैं, जो आपके browser में covariance matrix से निकाले गए हैं। तीर की लंबाई √λ के साथ बढ़ती है।

这是一个由 90 个点组成的真实三维点云。三个箭头是它真实的主成分，由你的浏览器根据协方差矩阵现场计算。箭头长度随 √λ 增长。

Scree Plot — Choosing How Many Components to KeepScree Plot — कितने Components रखें碎石图 — 选择保留多少个主成分

Python Implementation — Step by StepPython Implementation — Step by StepPython 实现 — 逐步讲解

Step 1: PCA from Scratch with NumPyStep 1: NumPy से शुरू से PCA第 1 步：用 NumPy 从零实现 PCA

import numpy as np
np.set_printoptions(suppress=True)   # plain decimals, no 1e-04

# ── Toy dataset: 6 students × 4 exam scores ──────────────────
X = np.array([
    [85, 88, 82, 90],
    [70, 72, 68, 74],
    [92, 95, 91, 96],
    [60, 58, 63, 55],
    [78, 80, 76, 82],
    [55, 52, 57, 50],
], dtype=float)

# Step 1 — Standardise (zero mean, unit variance)
mean = X.mean(axis=0)
std = X.std(axis=0, ddof=1)
X_std = (X - mean) / std

# Step 2 — Covariance matrix (of standardised data = correlation matrix)
C = np.cov(X_std.T)
print("Covariance matrix shape:", C.shape)
print(np.round(C, 3))

# Step 3 — Eigendecomposition (eigh: for symmetric matrices)
eigenvalues, eigenvectors = np.linalg.eigh(C)
idx = np.argsort(eigenvalues)[::-1]          # largest first
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

# Step 4 — Explained variance ratio
evr = eigenvalues / eigenvalues.sum()
print("\nEigenvalues:         ", np.round(eigenvalues, 4))
print("Explained var ratio: ", np.round(evr, 4))
print("Cumulative EVR:      ", np.round(np.cumsum(evr), 4))

# Step 5 — Project onto the top-2 PCs
W = eigenvectors[:, :2]
Z = X_std @ W
print("\nProjected data (2 PCs):\n", np.round(Z, 3))

# Check: sklearn gives the same answer (it uses SVD internally)
from sklearn.decomposition import PCA
pca = PCA().fit(X_std)
print("\nsklearn eigenvalues:  ", np.round(pca.explained_variance_, 4))
print("Same directions?     ", np.allclose(np.abs(pca.components_), np.abs(eigenvectors.T)))

Step 2: sklearn PCA — The Production WayStep 2: sklearn PCA — Production का तरीका第 2 步：sklearn PCA — 生产环境的做法

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()
X, y = iris.data, iris.target
feature_names = iris.feature_names

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca_full = PCA(n_components=None)            # keep all 4 components
pca_full.fit(X_scaled)

print("Explained variance ratio per component:")
for i, ev in enumerate(pca_full.explained_variance_ratio_):
    print(f"  PC{i+1}: {ev:.4f}  ({ev*100:.2f}%)")
cum = np.cumsum(pca_full.explained_variance_ratio_)
print(f"\n95% variance threshold: {np.argmax(cum >= 0.95) + 1} components")

pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)

print(f"\nOriginal shape:  {X.shape}")
print(f"Reduced shape:   {X_2d.shape}")
print(f"Total variance retained: {pca_2d.explained_variance_ratio_.sum():.4f}")

print("\nComponent loadings (how much each feature contributes):")
for i, comp in enumerate(pca_2d.components_):
    print(f"  PC{i+1}:")
    for fname, loading in zip(feature_names, comp):
        print(f"    {fname:20s}: {loading:+.4f}")

Step 3: Choosing n_componentsStep 3: n_components चुनना第 3 步：选择 n_components

from sklearn.decomposition import PCA

# A float between 0 and 1 = "keep enough PCs for this much variance"
pca_95 = PCA(n_components=0.95)
X_95 = pca_95.fit_transform(X_scaled)
print(f"Components needed for 95% variance: {pca_95.n_components_}")

# 'mle' = Minka's automatic choice
pca_mle = PCA(n_components='mle')
X_mle = pca_mle.fit_transform(X_scaled)
print(f"MLE auto-selected components:       {pca_mle.n_components_}")

# How much do we lose? Go back to 4-D and measure the error
X_reconstructed = pca_2d.inverse_transform(X_2d)
mse = ((X_scaled - X_reconstructed) ** 2).mean()
print(f"Reconstruction MSE (2 PCs):         {mse:.6f}")
print(f"Lost variance share (1 - retained): {1 - pca_2d.explained_variance_ratio_.sum():.6f}")

Complete Real-World Example — Digits Datasetपूरा Real-World Example — Digits Dataset完整实战案例 — 手写数字数据集

The digits dataset has 1,797 handwritten digit images of 8×8 pixels, so 64 features. We split the data first, then fit the scaler and PCA on the training images only, and compare a classifier with and without PCA.

Digits dataset में 8×8 pixels की 1,797 हाथ से लिखी digit images हैं, यानी 64 features. हम पहले data split करते हैं, फिर scaler और PCA सिर्फ़ training images पर fit करते हैं, और PCA के साथ और बिना एक classifier की तुलना करते हैं।

手写数字数据集有 1,797 张 8×8 像素的图片，也就是 64 个特征。我们先划分数据，再只在训练图片上拟合缩放器和 PCA，然后比较使用和不使用 PCA 的分类器。

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

digits = load_digits()
X, y = digits.data, digits.target
print(f"Dataset: {X.shape[0]} images × {X.shape[1]} pixel features")

# Split FIRST, so the test images never influence the scaler or the PCA
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

for k in [2, 10, 20, 30]:
    pca = make_pipeline(StandardScaler(), PCA(n_components=k)).fit(X_train)
    kept = pca[-1].explained_variance_ratio_.sum()
    print(f"Variance retained ({k:2d} PCs): {kept:.4f}")

print()
for name, k in [("No PCA (64 features)", None), ("PCA 30 components", 30), ("PCA 10 components", 10)]:
    steps = [StandardScaler()] + ([PCA(n_components=k)] if k else []) + [LogisticRegression(max_iter=2000)]
    model = make_pipeline(*steps)
    model.fit(X_train, y_train)
    print(f"  {name:22s}: test accuracy {model.score(X_test, y_test):.4f}")

PCA in a Full Machine-Learning Pipelineपूरी Machine-Learning Pipeline में PCA完整机器学习流水线中的 PCA

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold
from sklearn.datasets import load_wine

wine = load_wine()
X, y = wine.data, wine.target            # 178 wines × 13 chemical features

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=0.95)),
    ('svm',    SVC(kernel='rbf')),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(pipe, X, y, cv=cv, scoring='accuracy')
print(f"With PCA (95%):  CV accuracy {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

no_pca = Pipeline([('scaler', StandardScaler()), ('svm', SVC(kernel='rbf'))])
base = cross_val_score(no_pca, X, y, cv=cv, scoring='accuracy')
print(f"Without PCA:     CV accuracy {base.mean():.4f} ± {base.std():.4f}")

param_grid = {
    'pca__n_components': [2, 5, 8, 0.90, 0.95, 0.99],
    'svm__C':            [0.1, 1.0, 10.0],
    'svm__gamma':        ['scale', 'auto'],
}

grid = GridSearchCV(pipe, param_grid, cv=cv, scoring='accuracy', n_jobs=-1)
grid.fit(X, y)

print(f"Best CV accuracy: {grid.best_score_:.4f}")
print(f"Best params:      {grid.best_params_}")
print(f"PCs used:         {grid.best_estimator_.named_steps['pca'].n_components_}")

Incremental PCA — When Data Doesn't Fit in RAMIncremental PCA — जब Data RAM में न आए增量 PCA — 数据大到内存放不下时

import numpy as np
from sklearn.decomposition import IncrementalPCA, PCA

# Pretend this is a huge file: 100,000 rows × 50 features with real structure
# (5 hidden factors + noise). In real life you would read it chunk by chunk.
rng = np.random.default_rng(42)
factors = rng.normal(size=(100_000, 5))
mixing = rng.normal(size=(5, 50))
X_large = factors @ mixing + 0.5 * rng.normal(size=(100_000, 50))

ipca = IncrementalPCA(n_components=10)
for chunk in np.array_split(X_large, 50):       # 50 chunks of 2,000 rows
    ipca.partial_fit(chunk)                     # only one chunk in memory

print("IncrementalPCA explained variance ratio (top 10):")
print(np.round(ipca.explained_variance_ratio_, 4))
print(f"Top-5 total: {ipca.explained_variance_ratio_[:5].sum():.4f}")

# Compare with ordinary PCA (which needs all rows in memory at once)
pca = PCA(n_components=10).fit(X_large)
print(f"Ordinary PCA top-5 total: {pca.explained_variance_ratio_[:5].sum():.4f}")
cos = np.abs(np.sum(ipca.components_[:5] * pca.components_[:5], axis=1))
print("Direction match |cos| for PC1–PC5:", np.round(cos, 4))

X_transformed = ipca.transform(X_large[:1000])
print(f"Transformed shape: {X_transformed.shape}")

PCA Diagram — The Full Data FlowPCA Diagram — पूरा Data FlowPCA 示意图 — 完整的数据流

Key Hyperparametersमुख्य Hyperparameters关键超参数

When to Use PCA — and When Not ToPCA कब इस्तेमाल करें — और कब नहीं什么时候用 PCA — 什么时候不用

PCA vs Other Dimensionality Reduction MethodsPCA vs दूसरे Dimensionality Reduction तरीकेPCA 与其他降维方法对比

Golden Rulesसुनहरे नियम黄金法则