I need to know if the number of hidden layers affect the performance and accuracy of a neural network.
In other words, does increasing the number of hidden layers of a neural network increase the accuracy but decrease the performance? Or there is no rule for that?
Ultimately, the best way to think about neural network performance and addition of hidden layers is in terms of three stages:
1) increasing performance
2) diminishing performance
3) negative performance
In other words, addition of hidden layers helps improve the model, but only up to a certain point, and further addition of layers can actually harm the model's performance.
Let's take an example. Suppose we wish to develop a neural network in Keras to predict car sales using a regression-based neural network.
We are setting up a Sequential model with 5 input layers, 5 hidden layers, and 1 output layer:
model = Sequential()
model.add(Dense(5, input_dim=5, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()
Then, we are generating our loss statistics (or degree of error indicated in the model).
model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
history=model.fit(X_train, y_train, epochs=150, batch_size=50, verbose=1, validation_split=0.2)
print(history.history.keys())
# "Loss"
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
Let's see what our loss functions look like:
5 hidden layer configuration
We can see that loss for the validation and training sets is minimized after roughly 25 epochs.
Now, let's try expanding the hidden layers and see what happens.
(5,8) hidden configuration
(8,12) hidden configuration
(100,100) hidden configuration
You can see that up to the point where we use an (8, 12) hidden layer configuration, the loss on our model continues to improve (i.e. be minimized).
However, when we look at the (100, 100) configuration, we can clearly see overfitting. The training loss is often lower than the validation loss, and there is no consistency in the loss reduction, as evidenced by the frequent large spikes in loss.
Ultimately, hidden layers can help improve the accuracy of a model but only up to a certain point. Determining how many hidden layers a model should have is as much an art as a science, and is highly dependent on the type of data you are analyzing.
As a general rule, the more parameters model have (any model), the more patterns it can learn, but also, the more likely it is to overfit. So if you ask about training set accuracy, the more would be always better, however you would need to learn it proportionally longer. With test set accuracy, adding more neurons (making the network "wider") or layers (making it "deeper") would be improving accuracy, but at some point the test set accuracy would start dropping, as the model would start overfitting to training set. The relation would however be less straightforward with the modern, complicated architectures (pooling layers, skip connections, batch normalizations, etc.) and heavy usage of regularization that is common in deep learning, so this isn't exactly linear.
