Linear Discriminant Analysis (LDA)
First Principles Question
PCA finds directions of maximum variance. What if you want directions that best separate the classes?
The Core Idea
LDA is PCA with class awareness. Instead of maximizing total variance, it maximizes the ratio of between-class scatter to within-class scatter. You want classes to be far apart from each other, and tight within themselves.
The Math
Within-class scatter: Sw = S₁ + S₂ + ...
Between-class scatter: Sb = Σ Nᵢ·(μᵢ - μ)(μᵢ - μ)ᵀ
Maximize: |Sb| / |Sw|
Solution: eigenvectors of Sw⁻¹·Sb
Max discriminants: min(n_classes - 1, n_features)
LDA vs PCA
| PCA | LDA | |
|---|---|---|
| Supervision | Unsupervised | Supervised |
| Optimizes | Total variance | Class separation |
| Uses labels | No | Yes |
| Max components | n_features | n_classes − 1 |
Assumptions
- Each class follows a Gaussian distribution (from Probability & Distributions)
- All classes share the same covariance matrix
Prerequisites
Linear Algebra · Probability & Distributions · Principal Component Analysis
Builds To
Classification pipelines (LDA as a classifier, not just reducer)
Content Ideas
Obsidian note: “LDA is PCA that cheats — it knows the class labels. Here’s why that matters.”
X post: “PCA and LDA both reduce dimensions. One is blind. One cheats. Here’s the difference.”
GitHub: pca-lda-from-scratch — side-by-side PCA vs LDA on breast cancer dataset