Bayer pattern & demosaicing

The Bayer pattern

The most common color filter array repeats a $2 \times 2$ block with one red, two green and one blue filter:

Half of all pixels measure green, a quarter red and a quarter blue. Green gets twice the samples because it carries most of the brightness we perceive: the M and L cones of the eye are most sensitive to green-yellow light. The luminance of linear light, which models this sensitivity, is $Y = 0.2126\,R + 0.7152\,G + 0.0722\,B$ for the sRGB primaries: green has by far the largest weight. Our visual system resolves fine detail mainly in brightness and much less in color, so it makes sense to spend the samples there.

The raw image has a single value per pixel. Shown in gray it looks like a normal black-and-white image with a fine checkerboard texture wherever the scene is colored.

Demosaicing

Every pixel knows one channel and has to estimate the other two. Nearest neighbor copies them from the same $2 \times 2$ block. Bilinear interpolation averages the closest samples of the missing channel: the 4 direct neighbors for green, and 2 or 4 neighbors for red and blue. Written as a convolution of each sparse channel (zero where it was not measured):

At a measured pixel only the center weight hits a sample, so the value is kept. Elsewhere the kernel picks up exactly the neighbors described above. At the image border only the available samples are averaged.

Artifacts

• Zipper effect: along sharp edges, interpolation mixes values from both sides, in a pattern that alternates from pixel to pixel.
• False color: red and blue are sampled only at every other pixel in each direction. Fine gray detail, such as thin lines, is then interpolated differently in each channel and shows up in color, as colored moiré.

Real cameras use edge-aware demosaicing that interpolates along edges instead of across them, and many place a slightly blurring optical filter in front of the sensor to remove detail that the mosaic cannot capture.

Try this

• Zoom to 8× on a color patch of the test chart. Each pixel is red, green or blue, and green forms a checkerboard.
• Turn off “raw in color”. This is what the sensor actually stores: one number per pixel.
• Select a pixel under a red filter and check the green formula: the average of its four neighbors.
• Choose the zone plate, which is pure gray. The demosaiced image is full of colored rings, because red and blue alias before green does.
• Look at the line groups of the test chart with nearest neighbor, then with bilinear, and compare the PSNR values.
• Switch the right view to the error image and see where the errors are: at edges and fine detail, never in flat areas.
• Pick the colored text. Red text on blue has almost no green, so its edges are rebuilt from only a quarter of the pixels.