Five Sigma: How Physicists Claim a Discovery

A detector counts collision events. Background alone gives a bell curve of counts; a new particle would push the count higher. Physicists claim a discovery only when the count sits 5 standard deviations above the background, a false-alarm chance of about 1 in 3.5 million.

How it works

Hypothesis tests, p-values, Type I and II errors, and power

In one sentence

A test of $H_0$ against $H_1$ rejects $H_0$ when the test statistic falls in a rejection region chosen so that $P(\text{reject } H_0 \mid H_0) = \alpha$ . The p-value is the probability under $H_0$ of a statistic at least as extreme as the one observed. $\beta = P(\text{fail to reject } H_0 \mid H_1)$ , and the power is $1-\beta$ .

General formula

{\htmlData{sym=0}{z}} = \frac{{\htmlData{sym=1}{x}} - {\htmlData{sym=2}{\mu_0}}}{\htmlData{sym=3}{{\sigma}}}, \qquad {\htmlData{sym=4}{p}} = P(Z \ge {\htmlData{sym=0}{z}} \mid {\htmlData{sym=5}{H_0}}), \qquad {\htmlData{sym=6}{\alpha}} = P(\text{reject } {\htmlData{sym=5}{H_0}} \mid {\htmlData{sym=5}{H_0}}), \qquad {\htmlData{sym=7}{\beta}} = P(\text{keep } {\htmlData{sym=5}{H_0}} \mid {\htmlData{sym=8}{H_1}})

A p-value asks: if nothing were there, how surprising would this be? A tiny p-value means the data would be a very rare fluke under the null hypothesis.

What the symbols mean

$z$
Test statistic
How many standard deviations the count sits above the background.
$x$
Observed count
The number of collision events recorded.
$\mu_0$
Background mean
The count expected if there is no new particle.
$\sigma$
Background spread
The standard deviation of the background count.
$p$
p-value
The chance of a count at least this high from background alone.
$H_0$
Null hypothesis
Nothing new: the count is just background.
$\alpha$
Type I error rate
The chance of claiming a discovery when there's nothing there.
$\beta$
Type II error rate
The chance of missing a particle that is really there.
$H_1$
Alternative hypothesis
A new particle adds events on top of the background.

The null hypothesis $H_0$ : there's no new particle, so the count $x$ is background with mean $\mu_0$ and standard deviation $\sigma$ . The alternative $H_1$ : a signal adds events.
Standardize the count, $z = (x - \mu_0)/\sigma$ . The p-value is the chance of a count at least this high if $H_0$ were true.
Reject $H_0$ when $z$ passes a threshold. That fixes the Type I error rate $\alpha$ (a false discovery). The Type II error rate $\beta$ is the chance of missing a real signal, and $1 - \beta$ is the power.
Raising the threshold makes false discoveries rarer but real signals harder to confirm. Particle physicists use 5σ partly because they search in many places at once, and false alarms add up.

Learn the math

Hypotheses and ErrorsStats · Lesson 28, Hypothesis Testing p-Values and the z-Test for a MeanStats · Lesson 29, Hypothesis Testing