Does increasing number of hidden layers improve accuracy or after adding some layers improvement won't happen? Does increasing number if hidden layers much more than the sufficient number of layers cause accuracy to decrease? Do we need more epoch to train network?


I recommend you taking a look at http://www.deeplearningbook.org/ , they explain really well the concept of "Capacity" (Chapter 5, page 110), which might give you some answers to your questions. I'll try my best though, for the sake of my answer.

1) Increasing the number of hidden layers might improve the accuracy or might not, it really depends on the complexity of the problem that you are trying to solve.

2) Increasing the number of hidden layers much more than the sufficient number of layers will cause accuracy in the test set to decrease, yes. It will cause your network to overfit to the training set, that is, it will learn the training data, but it won't be able to generalize to new unseen data. A picture taken from the aforementioned book gives a pretty good intuition for this concept enter image description here Where in the left picture they try to fit a linear function to the data. This function is not complex enough to correctly represent the data, and it suffers from a bias(underfitting) problem. In the middle picture, the model has the appropriate complexity to accurately represent the data and to generalize, since it has learned the trend that this data follows (the data was synthetically created and has an inverted parabola shape). In the right picture, the model fits to the data, but it overfits to it, it hasn't learnt the trend and thus it is not able to generalize to new data.

3) The number of epochs ... I am actually not sure if you necessarily need more epochs the more hidden layers that you have. I guess it depends on other factors such as regularization. If you are trying to solve a super simple problem, then if you have a shallow and a really deep network, and you train both for the same number of epochs, you would probably get a better test accuracy in the shallow one (due to the overfitting of the deeper network that I mentioned above). However, if the problem is complex, then you might need to train the shallow network more epochs and apply regularization to achieve the same accuracy results as with the deep network. I am not an expert in this field, so don't take this last answer very serious.

I run an experiment to see the validation cost for two models (3 convolutional layers + 1 Fully connected + 1 Softmax output layer), the blue curve corresponds to the model having 64 hidden units in the FC layer and the green to the one having 128 hidden units in that same layer. As you can see, for the same number of epochs (x-axis), the overfitting starts to occur earlier for the model having 128 hidden units (having more capacity). This overfitting point can be seen as when the validation cost stops decreasing and starts to increase.

enter image description here

Check that book, it is awesome.


Not the answer you're looking for? Browse other questions tagged or ask your own question.