Linear Regression

No straight line passes through every data point, so pick the line whose squared misses add up to the least. Linear algebra finds it in a single step. Drag the points.

How it works

Least squares and projections

In one sentence

The least-squares line minimizes the sum of squared vertical errors; its coefficients solve the normal equations $X^\top X\boldsymbol\beta = X^\top\mathbf y$ , so $X\hat{\boldsymbol\beta}$ is the projection of $\mathbf y$ onto the column space of $X$ .

General formula

{\htmlData{sym=0}{\hat{\boldsymbol\beta}}} = \left({\htmlData{sym=1}{X^{\top}}}{\htmlData{sym=2}{X}}\right)^{-1}{\htmlData{sym=1}{X^{\top}}}{\htmlData{sym=3}{\mathbf{y}}}

$X^\top X$ and $X^\top\mathbf y$ summarize the data. Solving finds the one line whose errors have no part left in any direction the line could still move.

What the symbols mean

$\hat{\boldsymbol\beta}$
Best-fit coefficients
The intercept and slope of the line that fits the data best (least total squared error).
$X^{\top}$
Transpose of X
X flipped over its diagonal, so rows become columns.
$X$
Data matrix
One row per data point: a 1 (for the intercept) and the point's x value.
$\mathbf{y}$
Observed values
The measured y value of every data point.

Each point gives an equation $y_i \approx m x_i + b$ . Stacked together they form $X\boldsymbol\beta \approx \mathbf{y}$ .
The best fit projects $\mathbf{y}$ onto the column space of $X$ .
That projection is the solution of the normal equations $X^{\top}X\boldsymbol\beta = X^{\top}\mathbf{y}$ .

Learn the math

LinesAlgebra · Lesson 22, Graphing and Functions Augmented MatricesAlgebra · Lesson 46, Systems of Equations The Dot ProductCalc III · Lesson 3, Vectors The Matrix Equation Ax = bLin Alg · Lesson 4, Linear Equations in Linear Algebra Least-Squares ProblemsLin Alg · Lesson 35, Orthogonality and Least Squares Applications to Linear ModelsLin Alg · Lesson 36, Orthogonality and Least Squares Principal Component AnalysisLin Alg · Lesson 40, Symmetric Matrices and Quadratic Forms Scatter Plots and CorrelationStats · Lesson 37, Regression and Correlation The Least-Squares Regression LineStats · Lesson 38, Regression and Correlation Multiple RegressionEng Stats · Lesson 22, Regression for Engineering Linearizing Engineering ModelsEng Stats · Lesson 23, Regression for Engineering

Learn the math

LinesAlgebra · Lesson 22, Graphing and Functions

Augmented MatricesAlgebra · Lesson 46, Systems of Equations

The Dot ProductCalc III · Lesson 3, Vectors

The Matrix Equation Ax = bLin Alg · Lesson 4, Linear Equations in Linear Algebra

Least-Squares ProblemsLin Alg · Lesson 35, Orthogonality and Least Squares

Applications to Linear ModelsLin Alg · Lesson 36, Orthogonality and Least Squares

Principal Component AnalysisLin Alg · Lesson 40, Symmetric Matrices and Quadratic Forms

Scatter Plots and CorrelationStats · Lesson 37, Regression and Correlation

The Least-Squares Regression LineStats · Lesson 38, Regression and Correlation

Multiple RegressionEng Stats · Lesson 22, Regression for Engineering

Linearizing Engineering ModelsEng Stats · Lesson 23, Regression for Engineering