Vergence & rectification, interactive stereo demo

Vergence

In the simple stereo setup the optical axes are parallel and meet only at infinity. Human eyes instead turn towards the point they look at, the fixation point. With the eyes or cameras a distance $c$ apart and the angle $\theta$ between their axes, the fixation point lies at the distance

D = \frac{c}{2 \tan(\theta / 2)}

A large angle means a near point, a small angle a far one. The angle changes quickly only for near points, which is why vergence helps human depth perception only up to about 2–2.5 m; beyond that we rely on other cues such as motion parallax, perspective and occlusion.

The fixation point appears at the same position in both images, so its disparity is zero. A point nearer than the fixation point is seen further outward in each image and has a positive disparity $d = x_L - x_R > 0$ ; a farther point a negative one. In the top view, all points with zero disparity lie on the circle through both centers and the fixation point.

Turning the cameras inwards makes the parts of the scene that both see larger and allows a larger baseline, which improves the accuracy. But the image rows are no longer aligned: the epipolar line of a point is slanted, and its match lies above or below its row.

Rectification

Rectification restores the simple setup without moving the cameras. Each camera is turned virtually about its own center to a common orientation $R_\text{rect}$ : the $x$ axis along the baseline and both optical axes perpendicular to it. Both images are then reprojected onto one common image plane parallel to the baseline. A change of orientation alone maps pixels through a homography:

\mathbf{x}' \sim H_i\, \mathbf{x}, \qquad H_i = K\, R_\text{rect}\, R_i^\top K^{-1}

$K^{-1}$ turns a pixel into a viewing ray in camera coordinates, $R_i^\top$ into world coordinates, $R_\text{rect}$ into the rectified camera, and $K$ back into a pixel. Afterwards the epipoles lie at infinity, all epipolar lines are horizontal, and corresponding points share their row. The search for a match becomes a search along a scanline, and the depth follows from $Z = f\,c / d$ with only positive disparities.

The warp stretches the images into trapezoids, and the stretching grows with the angle between the views. Where the warped image is enlarged, the new pixels are interpolated from fewer original ones.

Try this

Compare the dashed row of the selected point with its epipolar line in the right image: in the verged images they differ, and the match lies on the epipolar line, not on the row. In the rectified images both coincide.
Set $\theta = 0$ : the cameras are parallel, the fixation point moves to infinity, and rectification changes nothing.
Increase $\theta$ until the fixation point lies on the box near the cameras: now almost every point is farther and has a negative disparity in the verged images, while all rectified disparities stay positive.
Switch to all epipolar lines: in the verged images they fan out towards the epipoles far to the sides; after rectification they are parallel.
Select a corner of the wall high up on the left and watch $y_L - y_R$ : it grows with the vergence angle, and rectification brings it back to 0.
Push $\theta$ to 30°: the rectified outlines become strongly slanted trapezoids. Large angles between the views make rectification distort the images more.