I designed a neural network for a regression task with the following parameters:

  • 42 inputs (I tried to reduce them with PCA but results are similar)
  • 300 hidden neurons with ReLu activation
  • 15 output neurons with tanh activation
  • batch size 20
  • RMSProp optimizer
  • mean square error for the cost function
  • 0.01 learning rate
  • 0.5 dropout in the training phase
  • 1000 epochs

At each epoch, the training set is fed to the model with small batches and the training error is computed. When the training is over, the training error's trend looks like this:

enter image description here

After the first decreasing phase, the error reaches a plateau. At this point, the training set has been processed several times. However, after a lot of epochs the error decreases again. How can I find the reason for this behaviour?

  • $\begingroup$ Simulated annealing may be another option for you. $\endgroup$ Commented Feb 13, 2017 at 12:42
  • $\begingroup$ Did it happen just once, or each time you train the network (with different random initial weights) you observe such a behaviour? How many observations are in the training set? $\endgroup$
    – user31264
    Commented Feb 13, 2017 at 13:12

1 Answer 1


Your training first reached a local minimum and got stuck there for a while before reaching the global minimum (or at least a much better local minimum). This is normal, albeit sometimes frustrating, as it means you can't really be sure you've reached the best set of weights for the neural network.

As to specifically how your training got free of the local minimum, that's down to how the optimiser traverses the neural network's weight hypersurface in this particular instance, which I would argue for a network of your complexity is unknowable. There are several known ways in which an optimisation can get 'stuck', e.g. where a hypersurface valley is roughly the same size as the step size, and the optimiser oscillates between the walls rather than following the valley floor.

I recommend playing around with this particular example I created in Tensorflow Playground. It doesn't follow the same path every time, but it's complicated enough that it should exhibit some false minima and oscillatory values. This ought to help you visualise what the neural network itself is actually fitting as it goes about the optimisation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.