When I examined many neural network architectures, I saw that hidden layers are always larger (having more neurons) than output layer. I have never thought about hidden layers narrower than output layer, but two days ago, one of my friends asked about this. I could not make sure that.
I think that, if hidden layers have fewer number of neurons than output layer, a bottleneck occurs. It is like pumping water into at first 2 cm radius of pipe, and then maybe 5 cm radius of pipe. The narrow one damps out the pressure and slows down the flow of water. When it goes to larger one, very settled and slow water continues to flow. Likewise, when we compress, for example, 400 features into 20 features with two consecutive hidden layers, and then try to enlarge it to 40 with an output layer to predict one of 40 different classes, what we find is the fact that we loss some much information during shrinkage of activations in hidden layers, and then mapping 20 to 40 does not retrieve that lost info and intuition, besides enlargement may create additional inefficiency. I made some experiments about this topic, and I am sharing simple results:
Last hidden layer: 128 Output Layer: 40 Accuracy: 82% Last hidden layer: 40 Output Layer: 40 Accuracy: 80% Last hidden layer: 30 Output layer: 40 Accuracy: 79% Last Hidden layer: 20 Output layer: 40 Accuracy: 70%
These are my thoughts. What do you think about this ?