Saddle Points in a Neural Network's Loss

Training a model means rolling downhill on a loss surface. At a flat spot the gradient is zero, but that spot could be a valley floor, a hilltop, or a saddle that is downhill in one direction and uphill in another. The second derivatives tell them apart.

Drag on the surface to move the point, or click a red marker to jump to it.

How it works

Relative extrema and the second-derivative test

In one sentence

If $\nabla f(a, b) = \mathbf 0$ and $f$ has continuous second partials near $(a, b)$ , let $D = f_{xx}f_{yy} - f_{xy}^2$ there: $D > 0$ and $f_{xx} > 0$ give a local minimum, $D > 0$ and $f_{xx} < 0$ a local maximum, $D < 0$ a saddle point, and $D = 0$ is inconclusive.

General formula

{\htmlData{sym=0}{D}} = {\htmlData{sym=1}{f_{xx}}}{\htmlData{sym=2}{f_{yy}}} - {\htmlData{sym=3}{f_{xy}}}^2:\quad {\htmlData{sym=0}{D}} > 0,\ {\htmlData{sym=1}{f_{xx}}} > 0 \Rightarrow \text{min};\quad {\htmlData{sym=0}{D}} > 0,\ {\htmlData{sym=1}{f_{xx}}} < 0 \Rightarrow \text{max};\quad {\htmlData{sym=0}{D}} < 0 \Rightarrow \text{saddle}

Near a flat spot the surface looks like a quadratic bowl, dome or saddle. $D$ multiplies the curvatures in the two directions (minus the twist): same signs give a bowl or dome, opposite signs a saddle.

What the symbols mean

$D$
Discriminant
The Hessian test number: positive means a bowl or a dome, negative means a saddle.
$f_{xx}$
Curvature in x
How the slope in x changes as x changes: positive is cupped up.
$f_{yy}$
Curvature in y
The same in the y direction.
$f_{xy}$
Twist
How the x-slope changes as you move in y; it tilts the bowl diagonally.

Find critical points by solving $\nabla f = \mathbf 0$ . For $f = x^3 - 3x + ay^2$ they are $(1, 0)$ and $(-1, 0)$ .
At each one compute the Hessian test $D = f_{xx}f_{yy} - f_{xy}^2$ . Here $f_{xx} = 6x$ , $f_{yy} = 2a$ and $f_{xy} = 0$ .
With $a > 0$ : at $(1, 0)$ , $D = 12a > 0$ and $f_{xx} > 0$ , a minimum; at $(-1, 0)$ , $D < 0$ , a saddle.
Flip the sign of $a$ and the roles change: the minimum becomes a saddle and the saddle becomes a maximum.

Learn the math

Higher Order Partial DerivativesCalc III · Lesson 21, Partial Derivatives Relative Minimums and MaximumsCalc III · Lesson 27, Applications of Partial Derivatives