Gradient Descent

Training a model means finding the lowest point on a loss surface. The gradient points uphill, so you take small steps the opposite way until nothing goes lower.

How it works

Partial derivatives and the gradient

In one sentence

Gradient descent iterates $\mathbf x_{k+1} = \mathbf x_k - \eta\nabla f(\mathbf x_k)$ ; with a small enough step $\eta$ on a smooth function, $f$ decreases every step and the iterates approach a critical point.

General formula

{\htmlData{sym=0}{\mathbf{x}_{k+1}}} = {\htmlData{sym=1}{\mathbf{x}_k}} - {\htmlData{sym=2}{\eta}}\,{\htmlData{sym=3}{\nabla f}}({\htmlData{sym=1}{\mathbf{x}_k}})

$\nabla f$ points uphill, so subtracting it walks downhill; $\eta$ is the stride length.

What the symbols mean

$\mathbf{x}_{k+1}$
Next position
Where the ball steps to next.
$\mathbf{x}_k$
Current position
Where the ball is after k steps.
$\eta$
Learning rate
The step size. Too big overshoots the valley; too small crawls.
$\nabla f$
Gradient
Points straight uphill on the surface. Subtracting it steps downhill.

$\nabla f = \langle f_x, f_y\rangle$ collects the partial derivatives and points in the direction of steepest increase.
Each step moves against the gradient, scaled by the learning rate $\eta$ . Too small crawls, too large overshoots and diverges.
Momentum keeps a running average of past gradients, which smooths out zig-zags in long narrow valleys.

Learn the math

The Chain RuleCalc I · Lesson 22, Derivatives Functions of Several VariablesCalc III · Lesson 9, Three-Dimensional Space Partial DerivativesCalc III · Lesson 19, Partial Derivatives The Chain RuleCalc III · Lesson 23, Partial Derivatives Directional Derivatives and the GradientCalc III · Lesson 24, Partial Derivatives Relative Minimums and MaximumsCalc III · Lesson 27, Applications of Partial Derivatives Quadratic FormsLin Alg · Lesson 38, Symmetric Matrices and Quadratic Forms

Learn the math

The Chain RuleCalc I · Lesson 22, Derivatives

Functions of Several VariablesCalc III · Lesson 9, Three-Dimensional Space

Partial DerivativesCalc III · Lesson 19, Partial Derivatives

The Chain RuleCalc III · Lesson 23, Partial Derivatives

Directional Derivatives and the GradientCalc III · Lesson 24, Partial Derivatives

Relative Minimums and MaximumsCalc III · Lesson 27, Applications of Partial Derivatives

Quadratic FormsLin Alg · Lesson 38, Symmetric Matrices and Quadratic Forms