The Least-Squares Regression Line

The regression line is simply the best straight line through a scatter plot, the one you'd use to predict $y$ from $x$. The odd name comes from Francis Galton, who noticed that tall fathers' sons tend to be less tall, "regressing" toward the average; the word stuck to the whole method. The formulas are short: $\hat y = a + bx$ (an ordinary line, with the hat meaning "predicted"), $b = r\frac{s_y}{s_x}$ (the slope, which is the correlation rescaled into real units) and $a = \bar y - b\bar x$ (the intercept, chosen so the line passes through the averages). Three small formulas, and you've got your line.

Least squares. For any line, the residuals $y - \hat y$ are the vertical misses. The regression line minimizes $\sum(y - \hat y)^2$. Why squared? For the same reason as in the standard deviation: misses above and below the line shouldn't cancel, and squaring also punishes big misses more than small ones. Calculus gives the slope $b = \frac{\sum(x - \bar x)(y - \bar y)}{\sum(x - \bar x)^2}$, which equals $r\frac{s_y}{s_x}$. There's a picture further down where each squared miss is drawn as an actual square.

Why $r\frac{s_y}{s_x}$? Think in z-scores, as in the last lesson. In z-score units the best line has slope exactly $r$: one standard deviation up in $x$ predicts $r$ standard deviations up in $y$. To convert back to real units, multiply by $s_y$ (to get $y$'s units) and divide by $s_x$ (to undo $x$'s). That's all the formula is.

Interpreting the slope: "each additional hour of study is associated with $b$ more points, on average." Associated, not caused, unless the data come from an experiment.

Regression toward the mean. Since $|r| \le 1$, an x value 2 standard deviations above average predicts a y value only $2|r|$ standard deviations from average. That's Galton's tall fathers: unless the correlation is perfect, extreme values of $x$ predict less extreme values of $y$.

A common mistake. It's tempting to read the intercept $a$ as meaningful every time. It's only the prediction at $x = 0$, and often $x = 0$ is far outside the data (a house with 0 square feet, a car with negative age). Then the intercept is just where the line happens to cross the axis.

Regression predicts house prices from size, crop yields from rainfall and sales from ad spending, and it's the first model every data scientist fits.