SVD visualized

The singular value decomposition

Every real $m \times n$ matrix $A$ can be written as

• $U$ ($m \times m$) is orthogonal; its columns $u_i$ are the left singular vectors,
• $\Sigma$ ($m \times n$) is diagonal with the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$,
• $V$ ($n \times n$) is orthogonal; its columns $v_i$ are the right singular vectors.

The singular values are the square roots of the eigenvalues of $A^\top A$ (and of $A A^\top$), and the $v_i$ are the eigenvectors of $A^\top A$. The value panel checks $\sigma_i^2 = \lambda_i$ for the current matrix. The larger a singular value, the more its pair $u_i, v_i$ contributes to $A$.

Rotate, scale, rotate

Read $A x = U\,(\Sigma\,(V^\top x))$ from right to left. $V^\top$ is orthogonal, so it only turns the plane: the unit circle stays a circle, and $v_1, v_2$ land on the axes $e_1, e_2$. $\Sigma$ stretches the axes by $\sigma_1$ and $\sigma_2$: the circle becomes an axis-aligned ellipse. $U$ turns this ellipse into place. So $A$ maps the unit circle to an ellipse with semi-axes $\sigma_1 u_1$ and $\sigma_2 u_2$, and

The panels use the usual math orientation, $x$ right and $y$ up. Singular vectors are only defined up to sign: flipping $v_i$ and $u_i$ together gives the same $A$. Here $V$ is always chosen as a pure rotation. When $\det A < 0$, $A$ flips the plane, and then $U$ contains a mirror: the small arrow on the circle runs clockwise in the last panel. $|\det A| = \sigma_1 \sigma_2$ is the area of the ellipse divided by $\pi$.

Solving $Ax = b$ with the pseudo-inverse

Undoing the three steps in reverse order gives the solution: rotate $b$ back with $U^\top$, divide by the singular values, rotate with $V$:

When a singular value is tiny, dividing by it blows up every error in $b$ along $u_i$. When it is zero, $A$ squashes the plane onto a line, most $b$ are not reachable, and the division is impossible. The pseudo-inverse replaces $1/\sigma_i$ by 0 for all $\sigma_i \le \tau$. The result is the least-squares solution, the $x$ that minimizes $|Ax - b|$, and among all of those the shortest one. In the last panel, the ring $Ax$ then sits at the point of the line closest to $b$. This works for any shape of $A$, including overdetermined systems with more equations than unknowns.

Rank-$p$ approximation

Written as a sum, the SVD builds $A$ from rank-1 layers $\sigma_k u_k v_k^\top$, one per singular value. Keeping only the first $p$ of them gives

It is the best approximation of rank $p$: no other matrix of rank $p$ is closer to $A$ in the sum of squared differences, and that sum is exactly $\sum_{k > p} \sigma_k^2$. A grayscale image is an $m \times n$ matrix, so it can be stored as $p$ columns of $U$ and $V$ plus $p$ singular values, $p\,(m + n + 1)$ numbers instead of $m n$. This is the dimensionality reduction from the lecture: the first singular vectors capture the dominant structure, the last ones mostly fine detail and noise.

Where it comes back

The same decomposition solves the homogeneous problem $Ax = 0$: among all unit vectors, $x = v_n$, the right singular vector of the smallest singular value, makes $|Ax|$ smallest, and $|A v_n| = \sigma_n$. This is the tool behind total least squares, estimating a homography and calibrating a camera.

Try this

• Choose Rotation: $\sigma_1 = \sigma_2 = 1$, the circle stays a circle and $\Sigma = I$. Then choose Scaling: now $U = V = I$ and only the middle step does anything.
• Choose Shear, $[1\ 1;\ 0\ 1]$: its singular values are $1.618$ and $0.618$, the golden ratio and its inverse. Their product is $\det A = 1$: a shear keeps the area.
• Choose Mirror: $\det A = -1$, the last step reads "rotate + mirror" and the small arrow on the ellipse turns clockwise.
• Choose Nearly singular and set $\tau = 0$. Click $b$ a little off the long axis of the ellipse: $1/\sigma_2 \approx 50$ sends $x$ far outside the first panel. Raise $\tau$ above $\sigma_2 \approx 0.02$: $x$ jumps back to a short vector, and $Ax$ (ring) moves onto the long axis, no longer exactly on $b$.
• Choose Singular: the ellipse collapses to a segment and $\sigma_2 = 0$. Wherever you click $b$, the ring $Ax$ lands at the closest point of the segment's line, and $x$ stays on the line through $\pm v_1$.
• Set $p = 1$: the image consists of copies of one row, each scaled by a different factor. Increase $p$ and watch the PSNR rise. Above $p = 68$, $p\,(m + n + 1)$ exceeds $m n = 19200$ and storing the SVD costs more than the image.
• Compare the images: the test chart, made of axis-aligned blocks, has singular values that drop steeply, so a small $p$ already looks sharp. The zone plate's circles need many more.