Distance & Similarity

First Principles Question

How do you measure how alike two things are when they’re represented as numbers?

The Core Idea

Every ML model implicitly or explicitly defines what “close” means. The choice of distance metric is a hidden assumption baked into the model. Change the metric, change what the model considers similar — and therefore change its predictions entirely.

Key Concepts to Cover

• Euclidean distance (L2) — straight-line distance; sensitive to scale
• Manhattan distance (L1) — sum of absolute differences; robust to outliers
• Cosine similarity — angle between vectors; ignores magnitude (useful for text)
• Minkowski distance — generalizes L1 and L2 with parameter p
• Why you must normalize features before computing distances
• Dot product as a similarity measure (projection)
• Loss functions as distance: MSE = L2 distance, MAE = L1 distance

Why It Matters for ML

• KNN: classifies by distance to nearest neighbors
• PCA: covariance matrix encodes how features co-vary (a form of similarity)
• SVM: margin maximization depends on distances to decision boundary
• CNN loss: MSE and cross-entropy are distance measures in output space
• Clustering: k-means, DBSCAN are entirely defined by a distance metric

Prerequisites

Linear Algebra

Builds To

Principal Component Analysis · Convolutional Neural Network · KNN · SVM (future notes)

Content Ideas

Obsidian note: “Every ML model secretly answers the question: what does ‘close’ mean?” X post: “MSE loss is just L2 distance. MAE loss is L1 distance. Your loss function is a ruler.” GitHub: Side-by-side comparison of KNN with L1 vs L2 vs cosine distance on same dataset