How Sure Is Your Model's Test Accuracy?

A model gets 12 of 100 test examples wrong. Its true error rate on new data is probably near 12%, but not exactly: a confidence interval says how far off it could be, and it shrinks only like 1/√n as the test set grows.

How it works

Confidence intervals for a proportion

In one sentence

For a sample proportion $\hat p$ from $n$ independent trials, the large-sample level- $C$ confidence interval for the true proportion is $\hat p \pm z^*\sqrt{\hat p(1-\hat p)/n}$ , where $z^*$ is the standard normal critical value with area $C$ between $-z^*$ and $z^*$ .

General formula

{\htmlData{sym=0}{\hat p}} \pm {\htmlData{sym=1}{z^*}}\sqrt{\frac{{\htmlData{sym=0}{\hat p}}(1-{\htmlData{sym=0}{\hat p}})}{\htmlData{sym=2}{{n}}}}

The test set is a sample of all the data the model will ever see. Bigger samples pin the true error down more tightly, but only at a square-root rate.

What the symbols mean

$\hat p$
Test error rate
The fraction of test examples the model got wrong.
$z^*$
Critical value
How many standard errors to reach out: 1.96 for 95% confidence.
$n$
Test-set size
How many examples the model was tested on.

Each test example is a trial: the model is wrong (1) or right (0). The observed error rate $\hat p$ estimates the true error rate on future data.
By the central limit theorem $\hat p$ is approximately normal with standard error $\sqrt{\hat p(1-\hat p)/n}$ , when the test set is large enough ( $n\hat p(1-\hat p) \ge 5$ or so).
The interval reaches $z^*$ standard errors either side of $\hat p$ : $z^* = 1.96$ for 95% confidence.
Quadrupling the test set only halves the interval: precision costs data. The test set must also be kept separate from the training data, or $\hat p$ comes out too optimistic.

Learn the math

The CLT for Sums and ProportionsStats · Lesson 24, The Central Limit Theorem Confidence Intervals for a ProportionStats · Lesson 27, Confidence Intervals Testing a ProportionStats · Lesson 31, Hypothesis Testing