Multiple Regression

Multiple regression is simply fitting a "line" that uses two or more inputs at once to predict an output, like weld strength from both current and weld time. The equation $\hat y = b_0 + b_1x_1 + b_2x_2$ is a straight-line fit with an extra term: $b_1$ (how much $y$ changes per unit of $x_1$, with $x_2$ held fixed) and $b_2$ (the same for $x_2$). That "held fixed" is the whole point, and it's what a one-input plot can't give you. The fitting is still least squares, just with a computer doing the algebra.

The scene. Spot welds hold car-body panels together, and their strength depends on the welding current and on the weld time. A plant's records hold both, along with the pull-test strength of coupons. Plot strength against current alone and the trend is weak: 0.2 kN per kA. But in those records, operators who turned the current up also shortened the weld time, to avoid spatter. The shorter times hid part of the current's benefit.

Holding the other fixed. The fitted plane $\hat y = b_0 + b_1x_1 + b_2x_2$ has $\partial\hat y/\partial x_1 = b_1$: slide along $x_1$ with $x_2$ frozen and the response changes by $b_1$ per unit. That's the question an engineer usually asks: if I raise the current and change nothing else, what happens?

Least squares in matrix form. Each run gives a row $[1\;\;x_1\;\;x_2]$ of the design matrix $X$. Minimizing $\|\mathbf y - X\mathbf b\|^2$ means the residual vector must be perpendicular to every column of $X$: $X^T(\mathbf y - X\mathbf b) = \mathbf 0$, the normal equations of least squares and linear models. The first equation says the residuals add to zero, so the plane passes through $(\bar x_1, \bar x_2, \bar y)$. Subtracting that point out leaves the 2 × 2 centered system above, small enough to solve by hand.

Why the one-variable slope misleads. Divide the first centered equation by $S_{11}$:

Here $c$ is the slope of $x_2$ on $x_1$ in the data. The simple slope is the partial slope plus a borrowed piece: $b_2$ times how much $x_2$ moves when $x_1$ does. In the weld records $c = -2$ cycles per kA, so the simple slope is $0.6 + 0.2(-2) = 0.2$. If $c$ is large enough the sign even flips (Simpson's paradox). Only when $S_{12} = 0$, as in a designed experiment, do simple and partial slopes agree, which is one big reason to design experiments.

Correlated inputs cost precision. Inverting the 2 × 2 system gives

with $s^2 = SSE/(n - 3)$. When the inputs move together, $r_{12}^2$ is near 1 and the data can barely tell their effects apart: the variance is inflated by $1/(1 - r_{12}^2)$, the variance inflation factor. For the weld records $r_{12} = -0.91$ and the factor is 6. To test a coefficient, compare $t = b_j/SE(b_j)$ with $t$ on $n - p$ degrees of freedom, as in testing against a spec.

Interactions. Sometimes the effect of one input depends on another: concrete gains strength faster when it cures warm. Add the product term $b_{12}x_1x_2$, and then $\partial\hat y/\partial x_1 = b_1 + b_{12}x_2$: the slope in $x_1$ changes with $x_2$. Read such a model carefully. On its own, $b_1$ is the slope in $x_1$ at $x_2 = 0$, which may lie far outside the data.

Adjusted $R^2$. Adding any predictor, even a column of random numbers, can only lower SSE, so $R^2$ never falls. Adjusted $R^2$ divides each sum of squares by its degrees of freedom, so it rises only when a new term reduces SSE by more than its share.

Process engineers fit multiple regressions to historical data from welding, molding, machining and chemical plants to find which settings drive quality, and the confounding in those records is the usual reason the findings must be confirmed by a designed experiment. Interaction terms appear wherever physics multiplies effects, from concrete maturity to battery aging, which speeds up with both temperature and state of charge. Adjusted $R^2$ and coefficient t tests decide which terms stay in a model.