Inner Product, Length and Orthogonality

The inner product is simply the dot product: multiply two vectors entry by entry, add up the results, and you get one number that tells you how much the two point the same way. It's called an "inner" product because, written as a matrix product, it's $\vec u^T\vec v$, a row times a column, which collapses to a single number (the "outer" product $\vec u\vec v^T$ blows up into a whole matrix instead). "Orthogonal" is just the formal word for perpendicular; it comes from the Greek for "right angle." What it's good for: length, distance and angle in any number of dimensions, even ones you can't draw. All we're really saying here is that $u_1v_1 + \cdots + u_nv_n$ (multiply matching entries and add) gives you everything, because $\|\vec v\| = \sqrt{\vec v\cdot\vec v}$ (square each entry, add, take the root: Pythagoras) and $\operatorname{dist}(\vec u, \vec v) = \|\vec u - \vec v\|$ (the length of the arrow from one point to the other). It's three small formulas built on one operation; bear with the notation.

Everything you know about dot products in 2D and 3D works in any dimension. You can't picture $\mathbb R^{100}$, but you don't need to: "perpendicular" there just means dot product zero.

Pythagoras: $\vec u \perp \vec v$ exactly when $\|\vec u + \vec v\|^2 = \|\vec u\|^2 + \|\vec v\|^2$. Think about why that works. Expanding gives $\|\vec u + \vec v\|^2 = (\vec u + \vec v)\cdot(\vec u + \vec v) = \|\vec u\|^2 + 2\,\vec u\cdot\vec v + \|\vec v\|^2$. The middle term is the only thing standing between you and the familiar $a^2 + b^2 = c^2$, and it disappears exactly when $\vec u\cdot\vec v = 0$.

Complements. $W^\perp$ collects every vector that's perpendicular to all of $W$ at once, and $\dim W + \dim W^\perp = n$. The big fact: the row space of $A$ and its null space are orthogonal complements, and so are the column space and the null space of $A^T$. It makes sense once you write out $A\vec x = \vec 0$ row by row: each entry of $A\vec x$ is a row of $A$ dotted with $\vec x$, so $A\vec x = \vec 0$ says exactly that $\vec x$ is perpendicular to every row.

A common mistake. It's tempting to think $W^\perp$ is everything in $\mathbb R^n$ that isn't in $W$, because "complement" sounds like "whatever's left over." It isn't: $W^\perp$ is only the vectors perpendicular to all of $W$, and it's a subspace in its own right (it contains $\vec 0$, for one thing). In $\mathbb R^3$, the complement of a line through the origin is the plane perpendicular to it, not every point off the line.

There's an animation further down that shows the dot product changing sign as one vector swings around another.

Recommendation engines and search engines measure similarity between users or documents as the angle between high-dimensional vectors (cosine similarity).