In most examples I've seen of CNNs with multiple two or more convolutional layers the second layer has more kernels (feature masks) than the second layer, usually around twice as many. My intuition is telling me that the reason for this is that the first layer is picking out features from the input data and the second layer is picking out subsequent combinations of those features in the first convolutional layer. If there are 5 different possible features in the data then there are many more combinations of ways these features could show up in the data. If I'm not wrong there is $\sum\limits_{n=1}^{5}{5 \choose n}$ combinations. So why do we not have that many kernels in the second layer? Is it because of memory limitations, or is it because the number of kernels is more related to the number of possible classes?


Usually between two such convolutional layers there is a pooling layer which reduces the size of each of these feature maps. So the deeper into the network you go the smaller the dimensions become. On the contrary, the number of feature maps is actually increased. This is done because we generally want a trade-off between high-resolution maps and more features.

Furthermore, the deeper you go in a network the more abstract the features it can detect. Say you have a CNN for facial recognition. The first layers would detect simple geometric shapes (lines, curves, etc.), the subsequent layers would detect more high-level features from these first ones (eyes, noses etc.) while the final layers would combine these to detect the highest level features (faces) but in low-resolution. So it makes sense we would want the model to detect many of these high-level features because they have more detail, while it wouldn't make sense wanting to detect many low level features (because in these layers the model understands just lines).

The above example can be seen in the image below:

  • $\begingroup$ Interesting, thank you for your answer! So is there any kind of relationship or rule of thumb for how many kernels you have in each layer? Also would you happen to have a source for those images? I'm writing my masters thesis and would like to look into that more. $\endgroup$ Apr 13 '18 at 14:58
  • $\begingroup$ One thing I tend to do is to double the number of filters and then use a pooling operation that cuts each dimension in half, so that after those two operations you require the half number of parameters than the previous layer. I haven't found, however, anywhere saying that this way is the best, it's just what I like to do. I think I got the image either from here or here. Search google images for "convolutional neural networks faces". $\endgroup$
    – user204007
    Apr 13 '18 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.