Training error in Deep Neural Network decreases after being stable for several epochs

Question

I designed a neural network for a regression task with the following parameters:

• 42 inputs (I tried to reduce them with PCA but results are similar)
• 300 hidden neurons with ReLu activation
• 15 output neurons with tanh activation
• batch size 20
• RMSProp optimizer
• mean square error for the cost function
• 0.01 learning rate
• 0.5 dropout in the training phase
• 1000 epochs

At each epoch, the training set is fed to the model with small batches and the training error is computed. When the training is over, the training error's trend looks like this:

After the first decreasing phase, the error reaches a plateau. At this point, the training set has been processed several times. However, after a lot of epochs the error decreases again. How can I find the reason for this behaviour?

Answer 1

Your training first reached a local minimum and got stuck there for a while before reaching the global minimum (or at least a much better local minimum). This is normal, albeit sometimes frustrating, as it means you can't really be sure you've reached the best set of weights for the neural network.

As to specifically how your training got free of the local minimum, that's down to how the optimiser traverses the neural network's weight hypersurface in this particular instance, which I would argue for a network of your complexity is unknowable. There are several known ways in which an optimisation can get 'stuck', e.g. where a hypersurface valley is roughly the same size as the step size, and the optimiser oscillates between the walls rather than following the valley floor.

I recommend playing around with this particular example I created in Tensorflow Playground. It doesn't follow the same path every time, but it's complicated enough that it should exhibit some false minima and oscillatory values. This ought to help you visualise what the neural network itself is actually fitting as it goes about the optimisation.