Information Theory
First Principles Question
How do you measure uncertainty? And how much does knowing one thing reduce your uncertainty about another?
The Core Idea
Entropy measures uncertainty. A fair coin flip has maximum entropy — you have no idea what’s coming. A biased coin that always lands heads has zero entropy — there’s nothing to learn. Information Gain measures how much a question reduces your entropy. This is the exact mechanism decision trees use to choose which feature to split on.
Key Concepts to Cover
- Entropy: H(S) = -Σ p·log₂(p) — average surprise in a distribution
- High entropy = high uncertainty, all outcomes equally likely
- Low entropy = low uncertainty, one outcome dominates
- Information Gain = entropy before split − weighted entropy after split
- Gini Impurity — a faster approximation of entropy for tree splits
- KL Divergence — how different two distributions are from each other
- Cross-entropy — the loss function used in classification models
Why It Matters for ML
- Decision Tree: picks the feature with highest information gain at each node
- Random Forest: same, across many trees on bootstrap samples
- GBT: cross-entropy is the loss function being minimized
- CNN classification: trained with cross-entropy loss
Prerequisites
Builds To
Decision Tree · Random Forest · Gradient Boosting Tree
Content Ideas
Obsidian note: “Entropy is just a measure of surprise. Here’s how it decides every split in a decision tree.” X post: “Decision trees aren’t magic. They just keep asking: which question reduces my confusion the most? That’s entropy.” GitHub: Visualize entropy and information gain on a toy dataset as a tree is built