Kernel sizes for multiple convolutional layer neural networks In most examples I've seen of CNNs with multiple two or more convolutional layers the second layer has more kernels (feature masks) than the second layer, usually around twice as many. My intuition is telling me that the reason for this is that the first layer is picking out features from the input data and the second layer is picking out subsequent combinations of those features in the first convolutional layer. If there are 5 different possible features in the data then there are many more combinations of ways these features could show up in the data. If I'm not wrong there is $\sum\limits_{n=1}^{5}{5 \choose n}$ combinations. So why do we not have that many kernels in the second layer? Is it because of memory limitations, or is it because the number of kernels is more related to the number of possible classes?
 A: Usually between two such convolutional layers there is a pooling layer which reduces the size of each of these feature maps. So the deeper into the network you go the smaller the dimensions become. On the contrary, the number of feature maps is actually increased. This is done because we generally want a trade-off between high-resolution maps and more features.
Furthermore, the deeper you go in a network the more abstract the features it can detect. Say you have a CNN for facial recognition. The first layers would detect simple geometric shapes (lines, curves, etc.), the subsequent layers would detect more high-level features from these first ones (eyes, noses etc.) while the final layers would combine these to detect the highest level features (faces) but in low-resolution. So it makes sense we would want the model to detect many of these high-level features because they have more detail, while it wouldn't make sense wanting to detect many low level features (because in these layers the model understands just lines).
The above example can be seen in the image below:

