In a given (including convolutional) network, suppose I have a pair of consecutive layers, the first with $x$ units, and the second with $y$ units, where $x\gt y$.
Will it be a problem if $(x-y)$ is large? For example, say, if $x\approx1000$, and $y\approx5$. To me, it intuitively seems so because it feels like a large amount of information is crammed into the units in the second layer, which might make it difficult to learn anything meaningful. However, I haven't seen it mentioned any where either supporting or rejecting this.
- Will the answer to this change based on where in the network this occurs? If so, how is it if it occurs in the early layers, middle layers, or at the end, perhaps with a large fully connected layer at the last but one layer and a softmax activated fully connected last layer for classification?
- Does it depend on the type of the layer, say, having a convolutional layer vs a fully connected layer?