Epipolar geometry: epipolar lines, epipoles & the fundamental matrix

Two cameras, one point

Two cameras with centers $C_0$ and $C_1$ look at the same scene. A 3D point $X$ appears in image 0 at $x_0$ and in image 1 at $x_1$ . The construction is symmetric, and all its parts lie in one plane, the epipolar plane through $C_0$ , $C_1$ and $X$ :

The baseline $B$ is the line through the two camera centers. It lies in every epipolar plane.
The epipoles $e_0$ and $e_1$ are where the baseline meets the image planes. $e_0$ is the image of $C_1$ in camera 0, and $e_1$ is the image of $C_0$ in camera 1.
The epipolar lines $l_0$ and $l_1$ are where the epipolar plane cuts the image planes. They contain $x_0$ and $x_1$ and the epipoles.

Every point $X$ has its own epipolar plane, but they all contain the baseline: they rotate about it like the pages of a book. So all epipolar lines of an image pass through its epipole, and together they form a pencil of lines.

Why the match has to lie on a line

The camera center is the right null space of the projection matrix, $P C = 0$ . If $X$ is on the ray through $C$ , so is every point $X + \lambda C$ , and they all project to the same pixel:

x \cong P (X + \lambda C) = P X + \lambda P C = P X

A pixel $x_0$ therefore only tells us the ray from $C_0$ through $x_0$ , not where $X$ is on it. Camera 1 sees this ray from the side, as a line: the epipolar line $l_1$ . Whatever depth $X$ has, $x_1$ lies on $l_1$ , so the search for the corresponding point shrinks from the whole image to a line. Once it is found, the two rays can be intersected, which gives the depth of $X$ .

Backprojection with the pseudo-inverse

$P_0$ is a 3 × 4 matrix and has no inverse, but its pseudo-inverse $P_0^+ = P_0^\top (P_0 P_0^\top)^{-1}$ gives a point with the same projection:

X^+ = P_0^+ x_0, \qquad P_0 X^+ = P_0 P_0^+ x_0 = x_0

$X^+$ lies on the ray, and together with the camera center it defines it: the ray is $\operatorname{join}(X^+, C_0)$ .

The fundamental matrix

Project both points of the ray into camera 1. $C_0$ goes to the epipole $e_1 = P_1 C_0$ , and $X^+$ goes to $P_1 P_0^+ x_0$ . In homogeneous coordinates the line through two points is their cross product, and the cross product with $e_1$ is a matrix product with the anti-symmetric matrix $[e_1]_\times$ :

l_1 = e_1 \times P_1 P_0^+ x_0 = \underbrace{[e_1]_\times P_1 P_0^+}_{F}\, x_0, \qquad [e]_\times = \begin{bmatrix} 0 & -e_2 & e_1 \\ e_2 & 0 & -e_0 \\ -e_1 & e_0 & 0 \end{bmatrix}

(here $e = (e_0, e_1, e_2)^\top$ are the three components of one epipole). The fundamental matrix $F$ is a linear map from points in image 0 to lines in image 1, and it contains the intrinsic and extrinsic parameters of both cameras. Since $x_1$ lies on $l_1$ :

l_1 = F x_0, \qquad x_1^\top F x_0 = 0, \qquad l_0 = F^\top x_1

This is the epipolar constraint. It holds for every pair of corresponding points, and it involves only pixels and $F$ , not the 3D point. That is what makes it possible to estimate $F$ from correspondences alone, with the 8-point algorithm.

$F$ is singular

$[e_1]_\times$ has rank 2, because $[e_1]_\times e_1 = e_1 \times e_1 = 0$ . So $F$ has rank 2 too, and $\det F = 0$ . Geometrically, $F$ maps a 2D family of points to a 1D family of lines, the pencil through $e_1$ : every line $F x_0$ contains $e_1$ , i.e. $e_1^\top F = 0$ . The epipole $e_0$ itself has no epipolar line, $F e_0 = 0$ : the ray through $e_0$ is the baseline, which camera 1 sees as the single point $e_1$ . Like every homogeneous matrix, $F$ is only defined up to scale; the panel scales it to $\|F\| = 1$ .

Special camera arrangements

If the cameras have the same orientation and are only shifted sideways, the baseline is parallel to both image planes. The epipoles move to infinity, and the epipolar lines become parallel and horizontal: a match is on the same image row. Stereo systems aim for exactly this, and rectification warps the images of any two cameras so that the epipolar lines become parallel to the image rows. If camera 1 moves straight forward, the epipole lies in the image, at the point that all epipolar lines radiate from: the focus of expansion of the motion.

Try this

Change the depth of $X$ : the point $x_1$ slides along $l_1$ , while $x_0$ stays put. The whole ray is one line in image 1.
Turn on lines of all corners: all epipolar lines of an image meet in its epipole, the image of the other camera center.
Click different points in image 0 and watch the epipolar plane in the 3D view turn about the baseline.
Choose Parallel: the epipoles are at infinity, and every epipolar line is the row of its point. Set up to 0.5 m: the lines tilt but stay parallel, because the baseline is still parallel to the image planes. Set forward to 0.3 m instead: now they meet, far outside the image.
Choose Forward motion: both epipoles are at the principal point, and the epipolar lines fan out from the center of the image.
Change $f_1$ : the epipolar lines in image 1 scale about the principal point, and $F$ changes, but the epipolar plane in 3D stays the same.
Move camera 1 until its center appears inside image 0: the epipole $e_0$ marks exactly where camera 1 is.

Two-view geometry & epipolar lines