What makes neural network a *convolutional* neural network?

Question

What is the difference between a Convolutional Neural Network (CNN) and an ordinary Neural Network (NN)? What does convolution mean in this context?

Answer 1

Starting from the Neural Network perspective:

I would say that the base Neural Network has all neurons interconnected between layers. The convolutional version simplifies this model using two hypotheses:

• meaningful features have a given size in the image.
• features are shift equivariant (shifted input leads to similarly shifted output), and may occur anywhere in the image.

The first asumption is expressed by setting to zero the weights leading to a hidden neuron, except for a region of interest/patch from the input.

Shift invariance is obtained by sharing the same weights across all the patches. In order to capture features anywhere in the image, it is simpler to pave the input with patches only slided by one pixel.

Those simplifications drastically reduce the number of parameters and lead to much simpler computations which 'happen' to take the form of a convolution, hence the C in CNN.

Note 1: the fixed feature size hypothesis is alleviated by the use of multiresolution and/or by using separate networks with different patch sizes.

Note 2: equivariance is usually not as useful as invariance, so the latter is often emulated with additional pooling layers.

Alternative approach

Before deep learning, a popular problem solving method was to extract features and feed them to a classifier. For images, the features were often extracted using expertly chosen filters such as Gabor filters/wavelets. On can view CNN as a parameterized filtering function, where parameters are trained using methods for Neural Networks

Answer 2

In short, local connectivity and parameter sharing (optional).

In terms of image data,
local connectivity says only neurons within a local region should be connected together, which basically assumes that pixels nearby are correlated, and pixels far apart are independent.

Parameter sharing means that the same set of parameters applies to different regions, which assumes that local patterns are shared across the whole image.

But global parameter sharing is not necessary when, for example, the images you have are all frontal faces, in which case you'll know high level patterns (say eyes, noses) would only appear around some certain region of the images.

A paper for reference: http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

Answer 3

The mathematical operation of convolution means to compute the product of two (continuous or discrete) functions over all possible shift-positions. The most simple example is to convolve a 1-dimensional vector ${\bf {\it x}}=(x_1,x_2,x_3,\dots,x_n)^T$ with a sampled Gaussian function (the Gaussian probability density function), ${\bf {\it y}}=(y_1,y_2,y_3,\dots,y_v)^T$. Practically this means to compute the summed dot-product, element-by-element: \begin{equation} c\left(\frac{v}{2}\right) = \sum_{i\,=-v/2, \; i \le 0}^{v/2} x_{i} \cdot \ y_{(i+v/2)} \end{equation} Now letting the running variable $z$ in $c(z),\; z = \frac{v}{2}$ run over the whole range of the vector ${\bf {\it x}}$ yields a vector of output values from the convolution, for each position.

In a 2-dimensional (gray-level) image, a convolution is performed by a sliding-window operation, where the window (the 2-d convolution kernel) is a $v \times v$ matrix.

When a neural network is used for convolution, a $v$-by-$v$ window of pixel values can be provided as input. In this way, the neural network can be trained to recognize objects of a certain size. Also a feature-based neural network can perform a convolution, when the feature vector is computed locally for each pixel coordinate. See Fig.1 in the reference: [M. Egmont-Petersen, E. Pelikan, Detection of bone tumours in radiographs using neural networks, Pattern Analysis and Applications 2(2) ,1999, 172-183].

Image-processing applications of neural networks have been reviewed in: [M. Egmont-Petersen, D. de Ridder, H. Handels. Image processing with neural networks - a review, Pattern Recognition, Vol. 35, No. 10, pp. 2279-2301, 2002]