I need to know if the number of hidden layers effect the performance and accuracy of a neural network, in other words does the increasing in the number of hidden layers of a neural network increase the accuracy but decrease the performance or there is no rule for that?
Thanks in advanced


Ultimately, the best way to think about neural network performance and addition of hidden layers is in terms of three stages:

1) increasing performance

2) diminishing performance

3) negative performance

In other words, addition of hidden layers helps improve the model, but only up to a certain point, and further addition of layers can actually harm the model's performance.

Let's take an example. Suppose we wish to develop a neural network in Keras to predict car sales using a regression-based neural network.

We are setting up a Sequential model with 5 input layers, 5 hidden layers, and 1 output layer:

model = Sequential()
model.add(Dense(5, input_dim=5, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, activation='linear'))

Then, we are generating our loss statistics (or degree of error indicated in the model).

model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
history=model.fit(X_train, y_train, epochs=150, batch_size=50,  verbose=1, validation_split=0.2)

# "Loss"
plt.title('model loss')
plt.legend(['train', 'validation'], loc='upper left')

Let's see what our loss functions look like:

5 hidden layer configuration

5 layers

We can see that loss for the validation and training sets is minimized after roughly 25 epochs.

Now, let's try expanding the hidden layers and see what happens.

(5,8) hidden configuration

5 8 hidden layers

(8,12) hidden configuration

8 12 hidden configuration

(100,100) hidden configuration

100 hidden layers

You can see that up to the point where we use an (8, 12) hidden layer configuration, the loss on our model continues to improve (i.e. be minimized).

However, when we look at the (100, 100) configuration, we can clearly see overfitting. The training loss is often lower than the validation loss, and there is no consistency in the loss reduction, as evidenced by the frequent large spikes in loss.

Ultimately, hidden layers can help improve the accuracy of a model but only up to a certain point. Determining how many hidden layers a model should have is as much an art as a science, and is highly dependent on the type of data you are analyzing.

  • $\begingroup$ Your initial statement about diminishing returns suggests that adding more and more hidden nodes will lead to smaller and smaller improvements in the output, but that more nodes won't hurt anything. Your last graph, however, suggests something worse - it's possible to have too many hidden nodes, to the point of actually decreasing performance on the validation set! $\endgroup$ – Nuclear Hoagie Dec 19 '18 at 15:23
  • $\begingroup$ @ Nuclear Wang, Thanks, maybe "diminishing returns" was the wrong term to use. As you pointed out, adding more nodes will indeed hurt performance, as the last graph demonstrates. $\endgroup$ – Michael Grogan Dec 19 '18 at 17:42

As a general rule, the more parameters model have (any model), the more patterns it can learn, but also, the more likely it is to overfit. So if you ask about training set accuracy, the more would be always better, however you would need to learn it proportionally longer. With test set accuracy, adding more neurons (making the network "wider") or layers (making it "deeper") would be improving accuracy, but at some point the test set accuracy would start dropping, as the model would start overfitting to training set. The relation would however be less straightforward with the modern, complicated architectures (pooling layers, skip connections, batch normalizations, etc.) and heavy usage of regularization that is common in deep learning, so this isn't exactly linear.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.