Convolution operator in CNN and how it differs from feed forward NN operation?

Question

I understand that the architecture of Convolutional Neural Networks (CNN) and Feed forward (FNN) are quite different. And that CNNs use pooling and filters of shared weights over a patch of the image. I am not so clear on the core convolution operator (1):

If anyone could link me to an explanation, I have looked at the Colah blog and Nielsen's online book, and I understand what it is doing but don't understand the convolution operator.

Also, it looks quite similar to the the core FNN function is there and difference? (2).

(1) convolution operator is $a^1 = \sigma(b + w*a^0)$ which is equivalent to:

$a^1 = \sigma(b + \sum^4_{l=0}\sum^4_{l=0}w_{l,m}a_{j+l,k+m})$

(2) feed forward operation:

$a^1_i = \sigma(\sum^n_{j=1}w_{ij}x_j +b_i)$

Many thanks

Sources: functions taken from: http://neuralnetworksanddeeplearning.com/chap6.html

Answer 1

I think CNNs are often talked about as putting squares on top of bigger squares with the "neural network" aspect hidden. They're definitely neural networks and can be drawn out.

Apply the filter to the upper left 2x2 array.

Apply the filter to the upper right 2x2 array.

Apply the filter to the bottom left 2x2 array.

Apply the filter to the bottom right 2x2 array.

Here is the entire layer, with the 3x3 input image mapping to four neurons for the four positions in the image where convolution occurs.

You can draw those four neurons in a square array if you plan to do additional convolutional layers. That doesn't make so much sense with a 2x2 output, but you're probably working with images that are bigger than 3x3.

I think that it's a useful exercise to draw out a simple example like this. Another useful exercise is to predict how many parameters there will be in a layer of 3x3 filtering (10 filters) over a 5x5 image. (The answer is 100: 9 for each of the ten filters, plus one bias term per filter, but bias terms typically get ignored when we draw out the "web" of a neural network. (However, bias terms are important!) Now try it for 3x4 filtering (20 filters) over a 7x5 image!)

To compare with a fully-connected layer, the fully-connected layer would have each pixel linked to each of the four neurons in the next layer, and each of those $36$ connections would have its own color to indicate the connection having its own weight, rather than sharing weights as is done in a convolutional layer.

Answer 2

The core idea about convolutional neural networks is that, contrary to fully-connected layers, instead to assigning different weights per each pixel of the picture (or something else), you have some kernel that is smaller then the input picture and slides through it. What follows, we apply same set of weights to different parts of the picture (so called weight sharing). By this we hope to detect same patterns in different parts of the image.

To illustrate this, let's look at one-dimensional kernel that slides through a vector (say, a sentence):

    g(x[0:2] * W + b) = z[0]
    /   |   \
 x[0] x[1] x[2] x[3] x[4] 

         g(x[1:3] * W + b) = z[1]
         /   |   \
 x[0] x[1] x[2] x[3] x[4] 

              g(x[2:4] * W + b) = z[2]
              /   |   \
 x[0] x[1] x[2] x[3] x[4] 

As you can see, we have an input vector of length five $\boldsymbol{x} = (x_0,x_1,\dots,x_4)$ and apply same set of three weights $\boldsymbol{w} = (w_0, w_1, w_2)$ and bias term $b$. The convolution kernel slides through the vector by applying same weights to each part of the vector and produces output vector of length three $\boldsymbol{z} = (z_0, z_1, z_2)$, where each $z_i = g(\boldsymbol{x}_{i:i+2} \cdot \boldsymbol{w} + b) = g(x_i w_0 + x_{i+1} w_1 + x_{i+2} w_2 + b)$.

So basically, it applies the same operator, but at smaller scale, going through the input tensor part-by-part while sharing the weights.

You can find this tutorial and recorded lectures by the Stanford CS231n staff helpful.