Applications to Linear Models

Main idea

Any model whose unknown parameters just get multiplied by known numbers and added up can be fit with the same least-squares recipe, even if its graph is curved. Put one row per data point in a design matrix $X$ and solve $X^TX\beta = X^T\vec y$ .

\vec y = {\htmlData{sym=0}{X}}{\htmlData{sym=1}{\beta}} + {\htmlData{sym=2}{\vec\epsilon}}, \qquad \hat{\htmlData{sym=1}{\beta}} = \left({\htmlData{sym=0}{X}}^T{\htmlData{sym=0}{X}}\right)^{-1}{\htmlData{sym=0}{X}}^T\vec y

What the symbols mean

$X$
Design matrix
One row per observation, one column per parameter.
$\beta$
Parameters
The coefficients being estimated.
$\vec\epsilon$
Residuals
The leftover errors least squares minimizes.

Explanation

A linear model is simply a formula for predicting one quantity from others in which the unknown numbers you're fitting, the parameters, only ever get multiplied by known things and added up. The "linear" is about the parameters, not the shape of the graph: $y = \beta_0 + \beta_1x + \beta_2x^2$ draws a parabola, but it's still a linear model, because each $\beta$ appears on its own, to the first power. What it's good for: one recipe fits lines, parabolas and models with several inputs, which is what regression software is doing under the hood. All we're really saying here is that $\vec y = X\beta + \vec\epsilon$ means "the data equals the model's predictions plus some errors," with $X$ holding the known numbers (one row per observation), $\beta$ the parameters and $\vec\epsilon$ the errors. And $\hat\beta = \left(X^TX\right)^{-1}X^T\vec y$ is just the normal equations from the last lesson, solved. It's the same idea with new letters.

Linear in the parameters, not in $x$ . Fitting $y = \beta_0 + \beta_1x + \beta_2x^2$ is still linear algebra: the row for an observation at $x$ is $[1\;\;x\;\;x^2]$ . Think about why that's fine. Once you plug in a data point, $x$ and $x^2$ are just known numbers, like $2$ and $4$ . The only unknowns are the $\beta$ 's, and they show up exactly the way the unknowns in $A\vec x = \vec b$ do.

Several variables: $y = \beta_0 + \beta_1x_1 + \beta_2x_2$ gives rows $[1\;\;x_1\;\;x_2]$ , with one column per parameter.

Transforming models. $y = ae^{bx}$ isn't linear in $a, b$ , but $\ln y = \ln a + bx$ is.

A common mistake. It's tempting to fit $y = ae^{bx}$ by giving $e^{bx}$ its own column, because that worked so well for $x^2$ . It can't work: $b$ is inside the exponential, so you don't know what numbers to put in that column until you already know $b$ . That's why you take logs first, which turns the model into a straight line in $\ln a$ and $b$ .

In practice, software solves this with QR factorization instead of forming $X^TX$ , for accuracy.

There's an animation further down that shows a curved model fit to stopping-distance data with these same normal equations.

See it

Stopping distances measured at five speeds, from 20 to 60 mph, with the least-squares parabola y = β₀ + β₁x + β₂x². The model is curved, but it's linear in β₀, β₁ and β₂, so the same normal equations XᵀXβ = Xᵀy find it, with each row of X being (1, x, x²). Here they give y = 0.05x² + 1.1x. Notice how fast the curve climbs: it predicts 42 ft at 20 mph but 246 ft at 60 mph, about six times as far for three times the speed.

Worked examples

Example 1

Fit $y = \beta x$ (a line through the origin) to $(1, 2)$ and $(2, 3)$ .

1
There's only one parameter, so $X$ is a single column of $x$ -values: $X = \binom12$ . Then $X^TX = 5$ and $X^T\vec y = 2 + 6 = 8$ , and the normal equation is $5\beta = 8$ .

$\beta = \frac85$

Practice

Type your answer and check it. Solve them all to complete the lesson.

Problem 1+5 XP

Give the design-matrix row for fitting $y = \beta_0 + \beta_1x + \beta_2x^2$ at $x = 2$ .

Problem 2+5 XP

Fit $y = \beta x$ to $(1, 2)$ and $(2, 3)$ .

Problem 3+5 XP

Fit $y = \beta_0 + \beta_1x$ to $(0, 1)$ , $(1, 3)$ , $(2, 5)$ . Give $\beta_0, \beta_1$ .

Problem 4+5 XP

Is fitting $y = ae^{bx}$ directly a linear least-squares problem in $a$ and $b$ ?

Problem 5+5 XP

How many columns does $X$ have for $y = \beta_0 + \beta_1x_1 + \beta_2x_2$ ?

Problem 6+5 XP

Hooke's law says a spring's force is proportional to its stretch: $F = kx$ . A lab measures $(x, F) = (1, 2.1)$ , $(2, 3.9)$ and $(3, 6.0)$ , in centimeters and newtons. Fit $F = kx$ by least squares: find $k$ .

Challenge problem+25 XP

The hardest one here. Get it, and you've got this lesson.

Fit a parabola $y = \beta_0 + \beta_1x + \beta_2x^2$ to $(-1, 2)$ , $(0, 0)$ , $(1, 1)$ and $(2, 4)$ by least squares. Give $\beta_0, \beta_1, \beta_2$ .

Where it's used

Economists, epidemiologists and engineers fit linear models daily; logistic regression and neural networks generalize the same design-matrix setup.

Linear RegressionLin Alg · Computer Science Logistic RegressionStats · Computer Science

Least-Squares Problems Diagonalization of Symmetric Matrices