How are filters and activation maps connected in Convolutional Neural Networks?

Question

How are the activation maps at a given layer connected to the filters for that layer? I'm not asking about how to do a convolutional operation between the filter and the activation map, I am asking about the type of connectivity these two have.

For example, say you wanted to do full connectivity. You have f number of filters and n number of activation maps in a given layer. You wold get f * n number of activation maps in the next layer, and the number of activation maps would increase with each new layer. This is the way I assume it is done.

Or you could say each filter is connected to just one activation map. In this case, the number of filters would equal the number of activation maps, and every layer would have the same number of filters and activation maps. This is the current architecture of my network, and it seems to learn fine.

The main source of my confusion is looking at diagrams of convnets I see online. Some of them have the "full connection" between filters and activation maps, such as this - In the first layer you have 4 activation maps, and presumably 2 filters. Each map is convolved with each filter, resulting in 8 maps in the next layer. Looks great.

But here we have an architecture that doesn't make sense to me - How do you go from 6 maps in the first layer to 16 in the 2nd? I can think of ways to get 16 maps from 6, but they wouldn't make any sense to do.

Answer 1

The second convolutional neural network (CNN) architecture you posted comes from this paper. In the paper the authors give a description of what happens between layers S2 and C3. Their explanation is not very clear though. I'd say that this CNN architecture is not 'standard', and it can be quite confusing as a first example for CNNs.

First of all, a clarification is needed on how feature maps are produced and what their relationship with filters is. A feature map is the result of the convolution of a filter with a feature map. Let's take the layers INPUT and C1 as an example. In the most common case, to get 6 feature maps of size $28 \times 28$ in layer C1 you need 6 filters of size $5 \times 5$ (the result of a 'valid' convolution of an image of size $M \times M$ with a filter of size $N \times N$, assuming $M \geq N$, has size $(M-N+1) \times (M-N+1)$. You could, however, produce 6 feature maps by combining feature maps produced by more or less than 6 filters (e.g. by summing them up). In the paper, nothing of the sort is implied though for layer C1.

What happens between layer S2 and layer C3 is the following. There are 16 feature maps in layer C3 produced from 6 feature maps in layer S2. The number of filters in layer C3 is indeed not obvious. In fact, from the architecture diagram only, one cannot judge what the exact number of filters that produce those 16 feature maps is. The authors of the paper provide the following table (page 8):

With the table they provide the following explanation (bottom of page 7):

Layer C3 is a convolutional layer with 16 feature maps. Each unit in each feature map is connected to several $5 \times 5$ neighborhoods at identical locations in a subset of S2's feature maps.

In the table the authors show that every feature map in layer C3 is produced by combining 3 or more feature maps (page 8):

The first six C3 feature maps take inputs from every contiguous subsets of three feature maps in S2. The next six take input from every contiguous subset of four. The next three take input from some discontinuous subsets of four. Finally, the last one takes input from all S2 feature maps.

Now, how many filters are there in layer C3? Unfortunately, they do not explain this. The two simplest possibilities would be:

1. There is one filter per S2 feature map per C3 feature map, i.e. there is no filter sharing between S2 feature maps associated with the same C3 feature map.
2. There is one filter per C3 feature map, which is shared across the (3 or more) feature maps of layer S2 that are combined.

In both cases, to 'combine' would mean that the results of convolution per S2 feature map group, would need to be combined to produced C3 feature maps. The authors do not specify how this is done, but addition is a common choice (see for example the animated gif near the middle of this page.

The authors give some additional information though, which can help us decipher the architecture. They say that 'layer C3 has 1,516 trainable parameters' (page 8). We can use this information to decide between cases (1) and (2) above.

In case (1) we have $(6 \times 3) + (9 \times 4) + (1 \times 6) = 60$ filters. The filter size is $(14-10+1) \times (14-10+1) = 5 \times 5$. The number of trainable parameters in this case would be $5 \times 5 \times 60 = 1,500$ trainable parameters. If we assume one bias unit per C3 feature map, we get $1,500 + 16 = 1,516$ parameters, which is what the authors say. For completeness, in case (2) we would have $(5 \times 5 \times 16) + 16 = 416$ parameters, which is not the case.

Therefore, if we look again at Table I above, there are 10 distinct C3 filters associated with each S2 feature map (thus 60 distinct filters in total).

The authors explain this type of choice:

Different feature maps [in layer C3] are forced to extract different (hopefully complementary) features because they get different sets of inputs.

I hope this clarifies the situation.

Answer 2

You are indeed correct that the value before the @ indicates the amount of filters, and not the amount of feature maps (although for the first convolutional layers these values coincide).

Regarding your last question: yes it does make sense to have every feature map at layer l connected to every filter at layer l+1. The sole reason for this is that this greatly increases the expression power of the network, as it has more ways (paths) to combine the feature maps which thus allow it to better distinguish whatever is on the input image.

At last I don't know if you are practicing your neural network skills by implementing them yourself, but if you just want to apply convolutional networks to a specific task then there are already several excellent neural network libraries such as Theano, Brainstorm, Caffe

Answer 3

When we slide (convolve) a filter over an input image, it creates a 2-dimensional feature map (or activation map), which shows the responses of that filter at every spatial position. So, each feature map (activation map) corresponds to a specific filter. Each element in the feature map is the result of activation of a specific neuron (it's value shows the degree to which the corresponding feature is present in the input image).

In Convolutional Nets we pass many filters over an input image, where each filter is representing a different feature of the image. For example, in early layers, a filter may detect a horizontal line, and another filter detects vertical line. Then , the next layer may be able to identify corners (by combining horizontal and vertical line).

Also, I would like to add the complete formula to compute the output size (when we have padding $P$ and stride $S>1$): Suppose the input size is $M \times M$ and the filter size is $N \times N$:

$$\text{Output Size} = \frac{M-N+2P}{S} + 1$$

For example, if the input size is $5 \times 5$, the filter size is $3 \times 3$, stride is 2 and Padding is 1, the output size will be 3.

Please note that the depth of the output volume is a hyperparameter, which corresponds to the number of filters, each learning to look for something different in the input.

Here is a great course from Stanford on CNN: https://cs231n.github.io/convolutional-networks/