Relative Minimums and Maximums

A relative maximum is simply a hilltop, a point higher than everything right around it, and a relative minimum is the bottom of a dip. ("Relative" because it only has to beat its neighbors, not the whole landscape.) Finding them is how you optimize anything that depends on two inputs, like profit from two products. The test quantity $D = f_{xx}f_{yy} - (f_{xy})^2$ looks intimidating, but it's built from three bend measurements: $f_{xx}$ (how the surface curves going east), $f_{yy}$ (how it curves going north) and $f_{xy}$ (a twist term). Its sign tells you whether both directions curve the same way or not. That's really all it's doing.

Critical points. In one variable, you found maxes and mins where $f'(x) = 0$, because the tangent line is flat at the top of a hill. Same idea here: at a peak or pit, the tangent plane is flat, so both slopes are zero. Solve $f_x = 0$ and $f_y = 0$ together.

A flat tangent plane isn't enough, though, just like $f'(x) = 0$ wasn't enough in one variable. In two variables there's a new possibility: a saddle point, which curves up in one direction and down in another, like a mountain pass. The tangent plane is flat there, but it's neither a peak nor a pit.

Sorting them out. In one variable you checked the sign of $f''$. Now there are several second partials, and the surface might bend up in one direction and down in another. Here's the reasoning behind $D$. If $f_{xx}$ and $f_{yy}$ have the same sign and the twist $f_{xy}$ is small, the surface bends the same way in both directions, and $f_{xx}f_{yy}$ beats $f_{xy}^2$, so $D > 0$. If they have opposite signs, $f_{xx}f_{yy}$ is negative and $D < 0$: a saddle. A large twist term can also make $D$ negative, because it can tilt the bending so that some diagonal direction curves the other way. When $D > 0$, the sign of $f_{xx}$ says whether it's a bowl or a dome.

The test at a critical point:

When $D = 0$ the test simply can't decide, just like $f'' = 0$ in one variable, and you'd have to look at the function more closely.

Training a neural network means descending a loss surface with millions of dimensions, where saddle points vastly outnumber true minima: the second-derivative ideas here explain why optimizers stall there.