Quadratic Forms

A quadratic form is simply a polynomial in which every term has degree exactly two, like $3x_1^2 + 4x_1x_2 + 5x_2^2$: squares and cross products, with no plain $x$ terms and no constant. "Quadratic" because of the degree two, and "form" is the old word for a polynomial whose terms all have the same degree. What it's good for: it's the multivariable version of $ax^2$, and it's how you tell whether a critical point is a minimum, a maximum or a saddle. All we're really saying here is that $\vec x^TA\vec x$ packs those coefficients into a symmetric matrix: $a$ and $c$ (the coefficients of the squares) go on the diagonal, and $b$ (half the cross-term coefficient) goes in both off-diagonal spots. Multiply it out and you get back $ax_1^2 + 2bx_1x_2 + cx_2^2$. It's a polynomial written as a matrix product, nothing more.

A common mistake. It's tempting to put the whole cross-term coefficient, like the $4$ in $4x_1x_2$, in each off-diagonal spot, because that's the number you see. It isn't right: multiplying out $\vec x^TA\vec x$ produces the cross term twice, once from each off-diagonal entry, so each spot gets half, here $2$.

Principal axes. Changing to the eigenvector basis removes the cross term: $Q = \lambda_1y_1^2 + \lambda_2y_2^2$. Think about why that works. $A$ is symmetric, so the last lesson gives $A = PDP^T$. Substitute $\vec x = P\vec y$ and you get $Q = \vec y^TP^TAP\vec y = \vec y^TD\vec y$, and a diagonal matrix has no off-diagonal entries to make cross terms. The level curves are ellipses or hyperbolas aligned with the eigenvectors.

Classification: once $Q = \lambda_1y_1^2 + \lambda_2y_2^2 + \cdots$, its shape is all about the signs of the eigenvalues.

• All eigenvalues positive: positive definite, $Q > 0$ for $\vec x \ne \vec 0$ (a bowl).
• All negative: negative definite (a dome).
• Mixed signs: indefinite (a saddle).

Constrained extremes. On the unit sphere $\|\vec x\| = 1$, the maximum of $Q$ is the largest eigenvalue and the minimum is the smallest. It makes sense from the same picture: $P$ preserves lengths, so $\vec y$ is a unit vector too, and $\lambda_1y_1^2 + \cdots + \lambda_ny_n^2$ with $y_1^2 + \cdots + y_n^2 = 1$ is a weighted average of the eigenvalues, which can't go above the largest or below the smallest.

There's an animation further down that shows a bowl turning into a saddle as one eigenvalue changes sign.

The second derivative test in several variables is a quadratic form classification (of the Hessian); energy in physics and risk in portfolio theory are quadratic forms.