Why a Bigger Poll Can't Fix a Biased One

A city of 400 residents votes on a new bus line. Poll a random sample and the estimate lands near the truth, closer as the sample grows. Poll only the people who are easy to reach and the estimate stays wrong, no matter how many you ask.

How it works

Populations, samples and sampling bias

In one sentence

A statistic estimates a population parameter: the sample proportion $\hat p$ estimates $p$ . For a simple random sample of size $n$ , $\hat p$ is unbiased with standard error $\sqrt{p(1-p)/n}$ . A sample drawn from a frame that leaves out part of the population can be biased, and that bias does not shrink as $n$ grows.

General formula

{\htmlData{sym=0}{\hat p}} = \frac{1}{\htmlData{sym=1}{{n}}}\sum_{k=1}^{\htmlData{sym=1}{{n}}} {\htmlData{sym=2}{x_k}} \;\approx\; {\htmlData{sym=3}{p}} \pm \sqrt{\frac{{\htmlData{sym=3}{p}}(1-{\htmlData{sym=3}{p}})}{\htmlData{sym=1}{{n}}}}

Random sampling makes the sample a miniature of the city. Leaving a group out tilts every sample the same way, so averaging more of them never cancels it.

What the symbols mean

$\hat p$
Sample proportion
The share of polled residents who said yes: the statistic.
$n$
Sample size
How many residents were asked.
$x_k$
One answer
1 if the kth person polled supports the bus line, 0 if not.
$p$
Population proportion
The true share of all 400 residents who support it: the parameter.

The population is all 400 residents, and the parameter is the share $p$ who support the bus line. The sample is the $n$ residents actually asked, and the statistic is their share $\hat p$ .
In a simple random sample every resident is equally likely to be asked, so $\hat p$ is right on average and its random error shrinks like $1/\sqrt{n}$ .
Here support falls with income. A sampling frame that reaches only the richest neighborhoods (a convenience sample) tilts every poll the same way: that is bias.
A bigger sample shrinks the random error but not the bias. A huge biased poll is precisely wrong.

Learn the math

Populations, Samples and VariablesStats · Lesson 1, Sampling and Data Sampling Methods and BiasStats · Lesson 2, Sampling and Data Confidence Intervals for a ProportionStats · Lesson 27, Confidence Intervals