Residuals and Diagnostics

Regression diagnostics are simply the checks you run after fitting a line to make sure you should believe it. You already met residuals and $r^2$ in the intro course; the new ideas here are about single points. Leverage $h_{ii}$ (how far a point's $x$ sits from the others, which decides how hard it can pull on the line) and Cook's distance (how much the whole fit would move if you deleted that one point). The formulas have a few pieces, but each one answers a plain question: does a line fit, and is any one point running the show?

The scene. A test engineer measures how far a spring deflects under loads of 1 to 5 N, and the points line up beautifully. Then she adds one test at 12 N, where the coils touch ("coil bind") and the spring can't deflect any further. With that one point the fitted slope drops by two-thirds. Nothing in the fitted equation warns you; the diagnostics do.

Residual plots: what the line missed. Plot the residuals $e_i = y_i - \hat y_i$ against $x$ (or against $\hat y$). A good fit leaves a patternless band around zero. Two patterns matter most:

• Curvature (a U or an arch): the relationship isn't straight. Add an $x^2$ term or transform the variables.
• Fanning (spread growing with $x$): the noise is proportional to the size of the reading, as in many flow meters and chemical assays. The line is still sensible, but $s$, the intervals and the tests are wrong. A log transform or weighted least squares fixes it.

A normal probability plot of the residuals checks the normality that t intervals rely on.

Why $SST = SSR + SSE$. Split each deviation: $y_i - \bar y = (\hat y_i - \bar y) + e_i$. Square and add. The cross term is $2\sum (\hat y_i - \bar y)e_i = 2b_1\sum (x_i - \bar x)e_i = 0$, because the normal equations force the residuals to be uncorrelated with $x$. So the total variation splits cleanly into explained plus unexplained, and $R^2$ is the explained share. For a line it equals $r^2$, the squared correlation.

$R^2$ is not a fit test. A curved relationship can have $R^2 = 0.98$ and still be the wrong model, and a fanning one can have a high $R^2$ with badly wrong intervals. In calibration, $R^2 = 0.9996$ says nothing about whether $s$ is small enough for the job.

Leverage: where the line is pinned. The fitted value at point $i$ is a weighted sum of all the responses:

The weight on its own response, $h_{ii}$, is its leverage. The leverages add to 2, one per coefficient, so the average is $2/n$, and $h_{ii} > 4/n$ is flagged as high.

Leverage hides errors. Let $\hat y_{(i)}$ be what the other points predict at $x_i$. If $y_i$ happened to equal $\hat y_{(i)}$, adding point $i$ wouldn't change the fit at all. And $\hat y_i$ moves by $h_{ii}$ for every unit $y_i$ moves. Together:

A point with $h_{ii} = 0.9$ drags the line 90% of the way to itself, so its residual shows only a tenth of its real miss. That's why residuals are standardized by $\sqrt{1 - h_{ii}}$, and why bad high-leverage points don't show up as big raw residuals.

Influence is leverage times surprise. Cook's distance multiplies the two: a point is influential when it is both far out in $x$ (large $h_{ii}$) and off the trend (large $r_i$). A far-out point that agrees with the rest is valuable: it pins the slope. Investigate influential points instead of deleting them. The 12 N test was telling the truth about coil bind, just not about the linear range.

Calibration labs, test departments and data teams look at residual plots before they believe any fit. Fanning residuals are routine in flow metering and chemical assays and drive the switch to log scales or weighted fits. Leverage and Cook's distance are standard regression output, used to catch the single test at an extreme setting, or the one mislabeled sample, that would otherwise set a design value by itself.