Statistics & Bias-Variance
First Principles Question
Why is a model that fits training data perfectly almost always wrong?
The Core Idea
Any model’s error has two sources: bias (it’s too simple to capture the pattern) and variance (it’s so sensitive to training data that it memorizes noise). These trade off. The goal of ML is not to minimize training error — it’s to minimize generalization error. This meta-principle explains why every regularization technique, ensemble method, and cross-validation strategy exists.
Key Concepts to Cover
- Bias: systematic error from wrong assumptions in the model
- Variance: sensitivity to fluctuations in training data
- Bias-variance tradeoff: reducing one increases the other
- Overfitting: low bias, high variance — memorizes training data
- Underfitting: high bias, low variance — too simple to learn
- Train/test split — why you must evaluate on unseen data
- Cross-validation — more reliable estimate of generalization error
- Regularization (L1, L2) — penalizes complexity to reduce variance
- Null hypothesis and p-values — deciding if a result is real or noise
Why It Matters for ML
- Random Forest: averaging many high-variance trees reduces overall variance
- GBT: boosting reduces bias by correcting previous errors
- Ridge/Lasso Regression: regularization directly controls the bias-variance tradeoff
- Every model: cross-validation is how you know if it’s actually working
Prerequisites
Builds To
Every model note. This is the meta-principle that sits above all of them.
Content Ideas
Obsidian note: “Why a perfect training fit is a red flag — the bias-variance tradeoff from first principles.” X post: “Your model got 100% accuracy on training data. Congratulations. It learned nothing.” GitHub: Visualize bias-variance as polynomial degree increases on a toy dataset