Disparity, depth & triangulation, interactive stereo demo

Correspondence and reconstruction

A stereo system has to solve two problems:

Correspondence: which pixel in the left image and which in the right image show the same scene point?
Reconstruction: given a correspondence and the geometry of the cameras, where is the point?

This demo is about the second problem and assumes the simplest setup: both optical axes are parallel, both image planes lie in the same plane, and the scanlines are aligned, $y_L = y_R$ . A point can then only correspond to a point on the same scanline, and its two pixels differ only in $x$ . Other setups can be brought into this form by rectification.

Depth from disparity

Put the left center of projection at the origin and the right one at a distance $T$ , the baseline, along $X$ . Similar triangles give for both cameras

\frac{x_L}{f} = \frac{X}{Z} \;\Rightarrow\; X = \frac{x_L Z}{f}, \qquad \frac{x_R}{f} = \frac{X - T}{Z} \;\Rightarrow\; X = \frac{x_R Z}{f} + T

Setting both equal and solving for $Z$ :

x_R Z = x_L Z - f\,T \quad\Rightarrow\quad Z = \frac{f\,T}{x_L - x_R} = \frac{f\,T}{d}

The disparity $d = x_L - x_R$ is inversely proportional to the depth: near points have a large disparity, and $d = 0$ means a point at infinity. Geometrically, $Z = fT/d$ is where the ray through $x_L$ meets the ray through $x_R$ .

Depth resolution and the baseline

Pixels have a width, so a pixel pair does not determine a point but a small area: the overlap of the two pixel wedges, shaded in the top view. Its depth extent follows from the derivative of $Z = fT/d$ :

\left| \frac{\partial Z}{\partial d} \right| = \frac{f\,T}{d^2} = \frac{Z^2}{f\,T}

The depth error for one pixel of disparity error grows with the square of the distance. A longer focal length or a longer baseline makes it smaller, so a small baseline gives a large depth error. A large baseline has its own problems: the search for correspondences covers more pixels, the two views look more different, and less of the scene is visible in both images, especially near the cameras.

Noise and wrong matches

Sensor noise, quantization and discretization shift the measured pixels, and each shift tilts a ray. Near the cameras, where the rays meet at a large angle, that barely moves the intersection; far away, where they are almost parallel, it moves it a lot. A wrong correspondence is worse: if the pixel of $P$ in the left image is matched with the pixel of $Q$ in the right image, the rays still meet, but at a point where there is nothing.

Try this

Drag $P$ away from the cameras and watch the shaded cell grow: at 4 m it spans less than a meter in depth, at 10 m several meters.
Halve and double the baseline $T$ : the cells and the curve in the plot shrink as $T$ grows. Then make $T$ large, about 1.5 m, and drag a point close to the cameras: it leaves the region both cameras see.
Lower the focal length to 30 px: a point at 10 m has a disparity of only 1 px, and the next possible value, 0 px, means infinity.
Add noise of 0.5–1 px and press New noise a few times: the reconstruction of the far point $Q$ jumps much more than that of $P$ .
Swap the correspondence: the two rays from $P$ and $Q$ cross at two points that do not exist. Move $Q$ behind $P$ : one of the swapped pairs may have $d \le 0$ and no intersection at all.