Naive Bayes

First Principles Question

Given what you know about a new data point, which class is most probable?

The Core Idea

Apply Bayes theorem. To classify, compute P(class | features) for each class and pick the highest. The “naive” part: assume all features are independent of each other given the class. This is almost never true. And yet Naive Bayes works surprisingly well, especially for text.

The Math

P(class | features) ∝ P(class) · P(f₁|class) · P(f₂|class) · ... · P(fₙ|class)

P(class)      = prior     — how common is this class?
P(fᵢ|class)  = likelihood — how probable is this feature given the class?
P(class|features) = posterior — what we want

The Naive Assumption

Features are conditionally independent given the class. This means:

P(f₁, f₂ | class) = P(f₁|class) · P(f₂|class)

In practice this rarely holds. But it makes the math tractable and the model fast.

Variants

• Gaussian NB: assumes features are normally distributed per class
• Multinomial NB: for word counts in text classification
• Bernoulli NB: for binary feature presence/absence

What Makes It Special

It’s the most transparent classifier on this list. Every prediction traces directly to Bayes theorem. No optimization, no matrix operations — just counting and multiplying probabilities.

Prerequisites

Probability & Distributions

Builds To

Hidden Markov Model (shares the probabilistic generative model perspective)

Content Ideas

Obsidian note: “The most honest model in ML. It’s literally just Bayes theorem applied to classification.” X post: “Naive Bayes makes an assumption that’s almost never true. It still works. Here’s why.” GitHub: ml-from-scratch — Gaussian NB from scratch, spam classifier demo