3
$\begingroup$

When I examined many neural network architectures, I saw that hidden layers are always larger (having more neurons) than output layer. I have never thought about hidden layers narrower than output layer, but two days ago, one of my friends asked about this. I could not make sure that.

I think that, if hidden layers have fewer number of neurons than output layer, a bottleneck occurs. It is like pumping water into at first 2 cm radius of pipe, and then maybe 5 cm radius of pipe. The narrow one damps out the pressure and slows down the flow of water. When it goes to larger one, very settled and slow water continues to flow. Likewise, when we compress, for example, 400 features into 20 features with two consecutive hidden layers, and then try to enlarge it to 40 with an output layer to predict one of 40 different classes, what we find is the fact that we loss some much information during shrinkage of activations in hidden layers, and then mapping 20 to 40 does not retrieve that lost info and intuition, besides enlargement may create additional inefficiency. I made some experiments about this topic, and I am sharing simple results:

Last hidden layer: 128
Output Layer: 40
Accuracy: 82%

Last hidden layer: 40
Output Layer: 40
Accuracy: 80%

Last hidden layer: 30
Output layer: 40
Accuracy: 79%

Last Hidden layer: 20
Output layer: 40
Accuracy: 70%

These are my thoughts. What do you think about this ?

$\endgroup$
3
$\begingroup$

It is not true that hidden layers are always larger. One example of networks that use hidden layers of smaller size than output, and input, are autoencoders (see Wikipedia, here, or here). The point of using autoencoders is dimensionality reduction. Thanks to autoencoders you can train a model that is able to compress your data to smaller representation, or use it for de-noising.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.