Is the New Model Really Better? A Paired Test

Two models are scored on the same 10 test folds. Some folds are simply harder, dragging both scores down together. Testing the fold-by-fold differences cancels that shared difficulty, so a small but consistent improvement stands out.

How it works

Matched pairs vs two independent samples

In one sentence

For paired observations with differences $d_i$ , $i = 1, \dots, k$ , the paired statistic $t = \bar d/(s_d/\sqrt{k})$ has $k-1$ degrees of freedom under $H_0: \mu_d = 0$ . The two-sample statistic $(\bar b - \bar a)/\sqrt{s_a^2/k + s_b^2/k}$ ignores the pairing, so variation between pairs inflates its standard error.

General formula

t = \frac{{\htmlData{sym=0}{\bar d}}}{{\htmlData{sym=1}{s_d}}/\sqrt{\htmlData{sym=2}{{k}}}}, \qquad {\htmlData{sym=3}{d_i}} = {\htmlData{sym=4}{b_i}} - {\htmlData{sym=5}{a_i}}

Compare each fold with itself. Whatever made a fold hard affects both models, so subtracting removes it.

What the symbols mean

$\bar d$
Mean difference
The average gain of model B over model A across the folds.
$s_d$
SD of the differences
How much the gain varies from fold to fold.
$k$
Number of folds
Ten: each fold is a pair of scores.
$d_i$
Difference on fold i
Model B's accuracy minus model A's on the same fold.
$b_i$
Model B's score
Model B's accuracy on fold i.
$a_i$
Model A's score
Model A's accuracy on fold i.

Cross-validation scores both models on the same $k = 10$ folds, so the scores come in pairs $(a_i, b_i)$ , one per fold.
A paired test works on the differences $d_i = b_i - a_i$ : their mean $\bar d$ over their standard error $s_d/\sqrt{k}$ gives $t$ with $k-1$ degrees of freedom.
Fold difficulty moves both scores together, so it cancels in each difference. An unpaired two-sample test leaves that variation in and buries the improvement in noise.
Pairing is valid only when the two measurements share a unit (a fold, a patient, a twin); for two separate groups, use the two-sample test.

Learn the math

Two Independent MeansStats · Lesson 32, Comparing Two Groups Matched PairsStats · Lesson 33, Comparing Two Groups Comparing Two ProcessesEng Stats · Lesson 18, Testing Engineering Claims