Is it a bad idea if the number of units in consecutive layers of a NN drops by a large amount?

Question

In a given (including convolutional) network, suppose I have a pair of consecutive layers, the first with $x$ units, and the second with $y$ units, where $x\gt y$.

Will it be a problem if $(x-y)$ is large? For example, say, if $x\approx1000$, and $y\approx5$. To me, it intuitively seems so because it feels like a large amount of information is crammed into the units in the second layer, which might make it difficult to learn anything meaningful. However, I haven't seen it mentioned any where either supporting or rejecting this.

1. Will the answer to this change based on where in the network this occurs? If so, how is it if it occurs in the early layers, middle layers, or at the end, perhaps with a large fully connected layer at the last but one layer and a softmax activated fully connected last layer for classification?
2. Does it depend on the type of the layer, say, having a convolutional layer vs a fully connected layer?

Thank you.

Answer 1

Will the answer to this change based on where in the network this occurs?

Definitely. In most neural networks, a layer with a large amount of nodes is followed by the output layer with (respectively) a small amount of nodes.

it intuitively seems so because it feels like a large amount of information is crammed into the units

Indeed, a large amount of information is crammed into just a few units. However, if there are no abstract relationships between the nodes in the large layer, then it isn't necessarily a problem that x-y is big. However, when you have very complicated/abstract data, you first want to create some abstractness from the large layer by following it with a medium layer. Afterwards you use the abstractness in the medium layer to create values for a small amount of nodes.

But it is hard to give an all-inclusive answer. You must understand that x-y should be related to the complexity of the data. If it is complex data, you introduce a small x-y, if it is less complex data, you increase x-y.

If you increase x-y, you decrease the amount of information you pull from x. If you decrease x-y, you create more abstract information from x.

Does it depend on the type of the layer, say, having a convolutional layer vs a fully connected layer?

Can't answer that, sorry :)