A/B Testing a Signup Button

Half your visitors see a blue button and half see green. Green converts 5.4% of visitors and blue 4.8%: real difference or luck? The two-proportion z-test compares the gap with the noise that random assignment alone would produce.

How it works

Comparing two proportions with a z-test

In one sentence

To test $H_0: p_A = p_B$ with independent samples, use $z = (\hat p_B - \hat p_A)\big/\sqrt{\hat p(1-\hat p)(1/n_A + 1/n_B)}$ with the pooled proportion $\hat p$ . Under $H_0$ , $z$ is approximately standard normal when each sample has at least about 10 successes and 10 failures.

General formula

z = \frac{{\htmlData{sym=0}{\hat p_B}} - {\htmlData{sym=1}{\hat p_A}}}{\sqrt{{\htmlData{sym=2}{\hat p}}(1-{\htmlData{sym=2}{\hat p}})\left(\frac{1}{{\htmlData{sym=3}{n_A}}} + \frac{1}{{\htmlData{sym=4}{n_B}}}\right)}}

Two noisy estimates can differ by chance. The test asks whether they differ by more than their combined wobble.

What the symbols mean

$\hat p_B$
Green's conversion rate
The fraction of visitors who signed up after seeing the green button.
$\hat p_A$
Blue's conversion rate
The fraction who signed up after seeing the blue button.
$\hat p$
Pooled rate
All signups divided by all visitors: the shared rate if the buttons don't matter.
$n_A$
Visitors shown blue
The size of group A.
$n_B$
Visitors shown green
The size of group B.

Randomly split the visitors so each version gets $n$ of them. The conversion rates $\hat p_A$ and $\hat p_B$ are sample proportions.
If the buttons made no difference ( $H_0: p_A = p_B$ ), both samples share one rate, best estimated by pooling: $\hat p$ is the total conversions over the total visitors.
The gap $\hat p_B - \hat p_A$ divided by its standard error under $H_0$ gives $z$ ; the p-value is the chance of a gap at least that large from random assignment alone.
Randomization is what makes the comparison fair: it balances everything else, from time of day to device, between the two groups.

Learn the math

Experimental Design and EthicsStats · Lesson 4, Sampling and Data Testing a ProportionStats · Lesson 31, Hypothesis Testing Two ProportionsStats · Lesson 34, Comparing Two Groups