k-Nearest Neighbors and Why Units Matter

k-nearest neighbors labels a new customer by a vote of the k most similar past customers. But “similar” depends on units: measure income in dollars and age in years, and a $1 difference counts as much as a year, so income swamps everything. Converting both to z-scores puts them on equal footing.

How it works

Standardizing with z-scores

In one sentence

The z-score of a value $x$ in data with mean $\bar x$ and standard deviation $s$ is $z = (x - \bar x)/s$ . Standardizing each feature gives it mean 0 and standard deviation 1, so Euclidean distance weights the features equally; k-nearest neighbors predicts the majority label among the $k$ training points closest to the query.

General formula

{\htmlData{sym=0}{z}} = \frac{{\htmlData{sym=1}{x}} - {\htmlData{sym=2}{\bar x}}}{\htmlData{sym=3}{{s}}}, \qquad {\htmlData{sym=4}{d}} = \sqrt{({\htmlData{sym=0}{z}}_1 - {\htmlData{sym=0}{z}}_1')^2 + ({\htmlData{sym=0}{z}}_2 - {\htmlData{sym=0}{z}}_2')^2}

A z-score answers 'how unusual is this value?' without units, which is exactly what makes different features comparable.

What the symbols mean

$z$
z-score
How many standard deviations a value sits from its feature's mean.
$x$
A raw value
An age in years or an income in dollars.
$\bar x$
Feature mean
The average of that feature over the training customers.
$s$
Feature standard deviation
The typical spread of that feature.
$d$
Distance
How far apart two customers are, combining both features.

k-nearest neighbors stores the training examples. To classify a new point it finds the $k$ closest by distance $d$ and takes a majority vote of their labels.
Distance adds up squared differences in each feature, so a feature measured in big numbers (dollars) dominates one measured in small numbers (years).
Standardizing replaces each value with its z-score $z = (x - \bar x)/s$ : how many standard deviations it sits from the mean. Every feature then has mean 0 and standard deviation 1.
After scaling, a one-standard-deviation difference in age counts as much as one in income, and the nearest neighbors are the customers who are actually alike.

Learn the math

Measures of SpreadStats · Lesson 7, Descriptive Statistics

k-Nearest Neighbors and Why Units Matter