Ultimately, the best way to think about neural network performance and addition of hidden layers is in terms of three stages:
1) increasing performance
2) diminishing performance
3) negative performance
In other words, addition of hidden layers helps improve the model, but only up to a certain point, and further addition of layers can actually harm the model's performance.
Let's take an example. Suppose we wish to develop a neural network in Keras to predict car sales using a regression-based neural network.
We are setting up a Sequential model with 5 input layers, 5 hidden layers, and 1 output layer:
model = Sequential()
model.add(Dense(5, input_dim=5, kernel_initializer='normal', activation='relu'))
Then, we are generating our loss statistics (or degree of error indicated in the model).
model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
history=model.fit(X_train, y_train, epochs=150, batch_size=50, verbose=1, validation_split=0.2)
plt.legend(['train', 'validation'], loc='upper left')
Let's see what our loss functions look like:
5 hidden layer configuration
We can see that loss for the validation and training sets is minimized after roughly 25 epochs.
Now, let's try expanding the hidden layers and see what happens.
(5,8) hidden configuration
(8,12) hidden configuration
(100,100) hidden configuration
You can see that up to the point where we use an (8, 12) hidden layer configuration, the loss on our model continues to improve (i.e. be minimized).
However, when we look at the (100, 100) configuration, we can clearly see overfitting. The training loss is often lower than the validation loss, and there is no consistency in the loss reduction, as evidenced by the frequent large spikes in loss.
Ultimately, hidden layers can help improve the accuracy of a model but only up to a certain point. Determining how many hidden layers a model should have is as much an art as a science, and is highly dependent on the type of data you are analyzing.