Convolution & correlation, interactive demo

Correlation

The filter $f$ , also called the kernel, has its origin in the center, so its offsets $k$ and $l$ run from $-r$ to $r$ for a $(2r + 1) \times (2r + 1)$ filter. Placing it over pixel $[m, n]$ of the image $I$ and adding up the products of weights and pixel values is called correlation:

h[m, n] = \sum_{k, l} f[k, l] \, I[m + k, n + l]

The same weights are used everywhere, so the filter is linear (filtering with $f_1 + f_2$ gives the sum of the two results) and shift invariant (filtering a shifted image gives the shifted result). Every operation with these two properties can be written this way.

Convolution

Convolution uses the filter flipped horizontally and vertically, i.e. rotated by 180°:

h[m, n] = \sum_{k, l} f[k, l] \, I[m - k, n - l], \qquad h = I * f

For symmetric filters such as the box or the Gaussian both are identical. For asymmetric ones they are not: a filter with a single 1 at $[k, l] = [2, 1]$ moves the image by 2 pixels left and 1 up under correlation, and right and down under convolution. The flip gives convolution its useful properties:

Commutative: $a * b = b * a$ , so there is no conceptual difference between filter and signal.
Associative: $a * (b * c) = (a * b) * c$ , so several filters in a row, $((a * b_1) * b_2) * b_3$ , can be combined into one filter $a * (b_1 * b_2 * b_3)$ . Correlation is not associative.
Distributes over addition, $a * (b + c) = (a * b) + (a * c)$ , and scalars factor out, $ka * b = a * kb = k(a * b)$ .
Identity: the unit impulse $e = [0, 0, 1, 0, 0]$ gives $a * e = a$ .

Convolving an image with a single bright pixel produces a copy of the filter, which is why $f$ is also called the impulse response.

Common filters

Box filter (mean filter): all $N$ weights are $1/N$ , so the result is the plain average of the neighborhood, $B = \frac{1}{N} \sum_{i=1}^{N} I_i$ .
Gaussian: weights that fall off with the distance from the center; a smoother blur than the box. The 5 × 5 preset uses binomial weights $\tfrac{1}{16}(1, 4, 6, 4, 1)$ in each direction.
Sharpen: the image plus four times the difference between each pixel and the average of its four neighbors, which boosts edges and fine detail.
Sobel: central differences in one direction combined with smoothing in the other, $S_x = \begin{bmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix}, \qquad S_y = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{bmatrix}.$ Their weights sum to 0, so flat areas give 0 and the output has positive and negative values. They are often scaled by $\tfrac{1}{8}$ to give the slope per pixel.

Filters that average have weights summing to 1, so the overall brightness of the image is preserved.

Boundary handling

Near the edge, part of the filter lies outside the image, and some rule has to supply those values. The left view shows the image extended by that rule:

Clip filter (black): pixels outside are 0. Averaging filters darken the border.
Wrap around: the image repeats periodically, so the opposite edge leaks in.
Copy edge: the edge pixels are replicated outward.
Reflect: the image is mirrored across the edge.

Applying a filter $n$ times acts like a single, $n$ times wider filter (associativity), so the border effects creep further into the image with every pass.

Try this

Select the shift filter, set “apply” to 20 and switch between correlation and convolution: the image moves in opposite directions.
With the shift filter, compare the border modes. Clip filter pads with black, wrap around brings in the opposite side of the image, copy edge smears the edge pixels.
Apply the 5 × 5 box 20 times with the clip filter: a dark frame grows from the edges. Copy edge and reflect keep the border bright.
Choose Sobel $S_x$ and switch between correlation and convolution: dark and light edges swap, because the flipped filter measures the slope with the opposite sign.
Select a pixel on an edge and check the values panel: the output is the sum of the element-wise products of the two matrices.
Set the result display to “absolute” with Sobel $S_x$ : now all vertical edges are bright, whatever their direction.
Emboss is asymmetric too: under convolution the light seems to come from the other side.
Edit the center weight of the identity filter to 2: the image gets twice as bright and clips. Then try a filter whose weights sum to 0.