Directional Derivatives and the Gradient

A directional derivative is simply the slope you feel if you walk across a hillside in some particular direction, not just due east or due north. The gradient is an arrow that points the steepest way uphill. The formula $D_{\vec u}f = \nabla f\cdot\vec u$ looks dense, but it's just two pieces: $\nabla f = \langle f_x, f_y\rangle$ (the slopes east and north, packed into one arrow) and $\vec u$ (the direction you're walking, as an arrow of length 1). Dot them together and you get your slope. It's not as bad as it looks; bear with it.

Partial derivatives only give the slope in two directions, along the axes. But you can walk across a hill at any angle. So what's the slope heading northeast?

Why a dot product. Take a small step of length $h$ in direction $\vec u = \langle a, b\rangle$. That changes $x$ by $ah$ and $y$ by $bh$. By the differential, $f$ changes by about $f_x(ah) + f_y(bh) = (f_x a + f_y b)h$. Divide by the step length $h$ to get the slope, and what's left, $f_x a + f_y b$, is exactly $\langle f_x, f_y\rangle\cdot\langle a, b\rangle$. That's why the gradient shows up.

What the gradient tells you. Now bring back the other face of the dot product: $\nabla f\cdot\vec u = \|\nabla f\|\|\vec u\|\cos\theta$, and $\|\vec u\| = 1$, so $D_{\vec u}f = \|\nabla f\|\cos\theta$, where $\theta$ is the angle between your direction and the gradient. That one formula tells you everything:

• It's largest when $\vec u$ points along $\nabla f$, where $\cos\theta = 1$. That's steepest ascent, with slope $\|\nabla f\|$.
• It's most negative in the opposite direction, steepest descent.
• It's zero when you walk perpendicular to $\nabla f$. Zero slope means the height isn't changing, so you're walking along a level curve.

Putting the last point another way, the gradient is always perpendicular to the level curves, pointing toward higher values. On a hiking map, it points straight across the contour lines, uphill.

A common mistake. It's tempting to plug in whatever direction vector you're given, like $\langle 3, 4\rangle$, because it already points the right way. But the derivation above assumed $\vec u$ has length $1$, and a longer vector inflates the answer by its length. Normalize first: with $\vec v = \langle 3, 4\rangle$, use $\vec u = \langle\frac35, \frac45\rangle$.

Gradient descent, the algorithm that trains nearly every machine learning model, repeatedly steps in the direction of $-\nabla f$. Water running downhill and heat flowing from hot to cold follow negative gradients too.