Measurement Error: Accuracy and Precision

Measurement error is simply the difference between what your instrument says and what's really there. All we're saying is that it comes in two flavors: a steady lean in one direction, and random jitter from reading to reading. The equation $y = x + \beta + \varepsilon$ is exactly that sentence: $y$ (what the gauge shows) equals $x$ (the true size) plus $\beta$ (the steady lean, called bias) plus $\varepsilon$ (the random jitter). The other formulas look like a wall of sigmas, but they all say one thing, that independent sources of scatter add as variances. It's a handful of simple ideas, so bear with the subscripts.

A caliper on a gauge block. You measure a 25.000 mm gauge block, a reference standard whose size is certified, five times with a digital caliper: 25.012, 25.008, 25.015, 25.010 and 25.005 mm. Two different things are going on. The readings average 25.010 mm, so the caliper reads 10 µm high on average: that steady offset is the bias. And the readings disagree with each other by several micrometers: that's random error.

So every reading is $y = x + \beta + \varepsilon$:

• Accuracy means small bias: on average the gauge tells the truth. Bias is fixed by calibration: measure a standard, then subtract the offset.
• Precision means small $\sigma_{\text{gauge}}$: readings agree with each other. Calibration can't fix scatter; only a better instrument or method can.

Picture a target. A tight cluster off to one side is precise but not accurate; a loose cluster around the bullseye is accurate but not precise.

Repeatability and reproducibility. Measure one part again and again with the same gauge and operator, and the scatter is repeatability. Let three operators do it, each with their own touch and fixture, and their averages differ a little: reproducibility. Together they are the gauge variation, and a gauge R&R study measures both.

Why variances add. Pick a part at random from production and measure it once. Its true size $X$ varies around the average $\mu$ with SD $\sigma_{\text{parts}}$, and the gauge adds an error $\varepsilon$ that doesn't care which part it's measuring. The bias only moves the average reading, so the reading's deviation from its own average is $(X - \mu) + \varepsilon$. Square it and average:

The middle term vanishes because $X$ and $\varepsilon$ are independent: $E[(X - \mu)\varepsilon] = E[X - \mu]\,E[\varepsilon] = 0 \cdot 0$. That's the variance rule from expected value, applied to a sum of independent pieces. Variances add; standard deviations don't. They combine like the legs of a right triangle,

so you can back out how much the parts really vary once a gauge study tells you $\sigma_{\text{gauge}}$.

Is the gauge good enough? A gauge study reports $\%\text{GRR}$, the gauge's share of the observed SD, and the precision-to-tolerance ratio $P/T$, how much of the tolerance band the gauge's ±3σ scatter covers. The usual industry guidelines: under 10% is good, 10% to 30% is marginal and over 30% means the gauge can't reliably tell good parts from bad ones.

The total error of one reading. Bias and scatter combine the same way: $E[(\beta + \varepsilon)^2] = \beta^2 + 2\beta E[\varepsilon] + E[\varepsilon^2] = \beta^2 + \sigma_{\text{gauge}}^2$, so the root-mean-square error is $\sqrt{\beta^2 + \sigma_{\text{gauge}}^2}$. Averaging $n$ readings shrinks the scatter to $\sigma_{\text{gauge}}/\sqrt n$, the standard error from the central limit theorem, but it leaves the bias exactly where it was.

Resolution. A display that shows 0.01 mm rounds every reading to the nearest $\Delta = 0.01$ mm, adding an error spread evenly over $[-\Delta/2, \Delta/2]$ with density $1/\Delta$. Its variance is a probability integral:

A common rule of thumb is the 10:1 rule: the resolution should be at most a tenth of the tolerance, or the gauge can't see the differences you care about.

A common mistake. When a gauge gives the same reading every time, it's tempting to call it accurate. It isn't necessarily. Agreement is precision; accuracy is about whether the readings center on the truth, and only a reference standard can tell you that. The same confusion drives two cousins of this mistake: thinking that averaging many readings removes bias (it only shrinks the scatter), and adding standard deviations instead of variances when combining sources of error.

Measurement error hides inside every number in this course. It inflates the spread of process data and makes capability indices look worse than the process really is. The variance-addition rule comes back for whole assemblies in tolerance stack-up.

Before a factory trusts any measurement of a part, it runs a gauge R&R study: a few operators measure the same parts repeatedly, and the study splits the observed variation into gauge and part pieces. Calibration labs keep instruments traceable to national standards, so a micrometer's bias is known and corrected. In chip making, where features are a few nanometers wide, metrology noise can eat a large slice of the tolerance, and engineers subtract gauge variance from process variance exactly as in this lesson to see what the process is really doing.