Are there any research papers documenting or formulating the behavior of training loss based on the complexity of the neural network?

Question

When I made the neural network more deeper and trained it for the same number of epochs, the training loss decreased but started increasing after a certain threshold. Shouldn't the training loss keep on decreasing and overfit the data? Is the relation of training or validation loss with the complexity of the network documented, like a theorem or formula.

edit- Trained for same epochs with the same training hyperparameters but the networks made deeper, the training loss decreases but begins to increase again after a threshold

Answer 1

I've never seen that papers, I don't think they are exists however I don't think anyone can ensure you that. Usually researches treat NN like black-box models with many dependencies.

Common belief is that optimization is working, so in long-distance view your network minimize loss function values. The main question how long loss is increasing and how much? Some optimization methods like momentum based force increasing loss values to get out from a local minima.

Behavior of your loss function is strongly depend not only of a model but also of data, but also from other reasons. It's hard to predict what is the main reason of this behavior based on your description. For example it could be exploding gradient problem:

If look at https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/autoencoder.ipynb You will see that autoencoder makes some noises on data, and adding more training epochs doesn't solve this problem. It could be ever worse! However if u change weight initialization this network will work properly. So in this case incorrect weight initialization was the reason.