Principal Component Analysis (PCA)
First Principles Question
If you have 64 features but most of them move together, can you summarize the data with fewer numbers without losing much?
The Core Idea
Find the directions in which your data varies the most. Project everything onto those directions. You’ve now compressed your data while keeping what matters. These directions are the eigenvectors of the covariance matrix. The eigenvalues tell you how much variance each direction captures.
The Math
1. Standardize data (zero mean, unit variance)
2. Compute covariance matrix: C = (1/n) XᵀX
3. Eigen-decompose C: C·v = λ·v
v = principal component (direction)
λ = variance explained in that direction
4. Sort by λ (descending), pick top k
5. Project: X_new = X · V_k
What PCA Does NOT Do
- It does not select features — it creates new ones (linear combinations)
- It is unsupervised — it doesn’t know or use class labels
- Accuracy can drop if too many components are removed
Key Insight from Your Notes
On the digits dataset (64 features): keeping 29 components retained 95% variance and nearly matched full accuracy (~96.9% vs 97.2%). Keeping only 2 components retained 27% variance and accuracy dropped to ~60%.
Prerequisites
Linear Algebra · Statistics & Bias-Variance
Builds To
Linear Discriminant Analysis — the supervised version of PCA
Content Ideas
Obsidian note: “PCA doesn’t select features. It creates new ones. Here’s the derivation.”
X post: “PCA doesn’t select features. It creates new ones. Most people get this wrong.”
GitHub: pca-lda-from-scratch — PCA in pure NumPy, visualize explained variance curve