JPEG & the DCT, interactive demo · Computer Vision Basics

The JPEG pipeline

A JPEG encoder converts the colors to YCbCr and usually stores the two chroma channels at lower resolution. Each channel is then processed the same way, which this demo shows for a gray image:

Split the image into 8×8 blocks and subtract 128, so that the values are centered on 0.
Transform every block with the discrete cosine transform (DCT).
Divide every coefficient by its step $Q(u, v)$ and round to an integer.
Read the integers in zigzag order and store them compactly with run-length and Huffman coding.

The decoder multiplies by $Q$ again and applies the inverse DCT. All steps can be undone exactly except the rounding in step 3: this is where the information, and the file size, is lost.

The discrete cosine transform

Like the Fourier transform, the DCT writes a block as a sum of waves, but it uses only cosines, and its coefficients are real numbers:

F(u, v) = \tfrac14 \, C(u) \, C(v) \sum_{x = 0}^{7} \sum_{y = 0}^{7} f(x, y) \cos\frac{(2x + 1) u \pi}{16} \cos\frac{(2y + 1) v \pi}{16}

with $C(0) = 1 / \sqrt 2$ and $C(k) = 1$ otherwise, which makes the transform orthonormal: it preserves the energy of the block, and the inverse uses the same cosines. $F(0, 0)$ is the DC coefficient, 8 times the mean of the shifted block; the other 63 are AC coefficients. The DCT behaves like a Fourier transform of the block mirrored at its borders. A mirrored block has no jump at the border, unlike a periodically repeated one, so smooth blocks need only a few low-frequency coefficients. For natural images, most of the energy ends up in the top-left corner: the DCT compacts the energy into few coefficients.

Quantization

Every coefficient is divided by a step size and rounded: $F_Q(u, v) = \operatorname{round}\big(F(u, v) / Q(u, v)\big)$ . The standard luminance table uses small steps for low frequencies and large steps for high frequencies, because the eye is much less sensitive to errors in fine detail. The quality setting scales the whole table: quality 50 uses it as is, quality 100 uses step 1 everywhere, and low qualities use steps up to 255. Most high-frequency coefficients then round to 0. The zigzag order visits the coefficients from low to high frequency, so the zeros collect at the end, and a single end-of-block code (EOB) replaces all of them.

The values panel estimates the storage with the entropy of the stored integers: the fewer different values and the more zeros, the fewer bits per coefficient an ideal code needs. It is a rough estimate; real JPEG also codes runs of zeros and the differences between neighboring DC coefficients.

Artifacts

Blocking: each block is quantized on its own. In smooth areas, only the DC coefficient survives, and every block becomes flat with a small step to its neighbor.
Ringing or mosquito noise: a sharp edge needs many high frequencies. Without them, the edge overshoots and ripples, like a truncated Fourier series, but only within the 8×8 block that contains the edge.
Loss of texture: fine repetitive detail, such as the fence, is smoothed away or turns into a coarser pattern.

Try this

Select a block in the sky and one on the fence: the smooth block keeps only a few coefficients, the fence needs many.
Lower the quality step by step and watch the stored coefficients turn black, from the bottom right towards the DC coefficient.
At quality 10, zoom in on the sky: the gradient turns into a staircase of flat blocks.
Zoom in on the text or the test chart at quality 20: ringing appears around every edge and stops at the block borders.
Look at the zigzag sequence of a block: at low quality it is a few numbers followed by EOB.
Switch to the flat table and adjust the quality until the entropy matches that of the JPEG table. Compare the PSNR, and look at where each table puts its errors.