Motion parallax & focus of expansion, interactive demo

Flow from a moving camera

Between two frames, the camera center moves by $\mathbf{T}$ and the camera turns by a small rotation $R_\delta$ , both given in the camera frame of the first image. A static point with camera coordinates $\mathbf{X}_c = Z\,K^{-1}\mathbf{x}_0$ in the first frame has the coordinates

\mathbf{X}'_c = R_\delta^\top (\mathbf{X}_c - \mathbf{T})

in the second, and its pixel moves from $\mathbf{x}_0$ to $\mathbf{x}_1 \sim K\mathbf{X}'_c$ . The arrows show this displacement, the motion field of the scene. Optical flow estimated from the images only approximates it (see patch-based vs regularized flow).

Motion parallax

If the camera moves parallel to the image plane ( $T_z = 0$ ) without turning, every point keeps its depth and its pixel moves by

\mathbf{x}_1 - \mathbf{x}_0 = -\frac{f}{Z} \begin{bmatrix} T_x \\ T_y \end{bmatrix}

All points move in the same direction, opposite to the camera, but the length is inversely proportional to the depth: near objects sweep across the view, the mountains barely move. This is motion parallax, the depth cue you see from a train window. In the plot, all points lie on the line through the origin. Turned around, the flow gives the depth, $Z = f\,\|\mathbf{T}\| / \|\mathbf{x}_1 - \mathbf{x}_0\|$ , as long as the motion of the camera is known.

This is also why a 2D translation is only a rough model for the motion of a whole image: it ignores parallax, and it only holds where the scene is at about the same depth.

Focus of expansion

With a forward component $T_z \neq 0$ , the image of the direction of travel,

\mathbf{e} = \frac{K\,\mathbf{T}}{T_z},

is the one point that does not move, and the translational flow of every other point lies on the line through it:

\mathbf{x}_1 - \mathbf{e} = \frac{Z}{Z - T_z}\,(\mathbf{x}_0 - \mathbf{e})

Moving forward, points stream away from $\mathbf{e}$ , the focus of expansion, and do so faster the nearer they are and the farther they are from $\mathbf{e}$ . An approaching object grows radially in the same way. Moving backward, the flow converges in a focus of contraction. So the flow tells where the camera is heading: a pilot landing a plane aims at the point the flow radiates from. The focus of expansion is the epipole of the first image: the image of the second camera center.

Rotation carries no depth

Turning the camera without moving it maps every pixel through the same homography,

\mathbf{x}_1 \sim K R_\delta^\top K^{-1} \mathbf{x}_0,

independent of $Z$ : near and far points move alike, and a rotation alone reveals nothing about depth (which is why panoramas are taken by turning around the camera center, see homography). It also distorts the image, e.g. a pan shifts the image and bends it slightly into a trapezoid. For small motions, the flow is approximately the sum of the translational part, which depends on depth, and the rotational part, which does not. With rotation, the total flow no longer radiates from $\mathbf{e}$ , and depth computed from the flow is wrong unless the rotation is removed first.

Try this

Choose sideways (train window) and press Play: the posts in front sweep by, the ground near the camera moves fastest, and the mountains stand still. In the plot, all points lie on one line.
Choose walking forward: the flow radiates from the focus of expansion on the horizon. Points near it barely move, even close ones; in the plot, the points no longer lie on a line, since the flow also depends on the distance to e.
Choose landing approach: as in the lecture's runway picture, the flow radiates from the point the camera is heading for, here on the ground ahead.
Choose forward and sideways, then move the translation slider $T_x$ : the focus of expansion moves sideways and leaves the image when the motion is mostly sideways.
Choose pan only: all points move by about the same amount regardless of their depth. Switch the flow to translation: there is nothing left.
Choose forward + pan and switch between total, translation and rotation: the total flow seems to radiate from a different point, but only the translational part radiates from e. Compare the two depth estimates of the selected point.
Set $T_z$ negative: the camera moves backward and the flow converges in a focus of contraction.