I am training a network to solve a regression problem using Keras. During training, the loss of my model goes directly from 7 to more than 300000 dramatically. Here is the training output: enter image description here

Here is the loss picture: enter image description here

I wonder what could be the cause of this problem? Thanks

  • $\begingroup$ What optimization algorithm are you using? And what are the hyper-parameters (step-size, momentum etc.) of the respective optimization algorithm? $\endgroup$ – Armen Aghajanyan Feb 29 '16 at 18:24
  • $\begingroup$ Sorry I forgot that, I use Adam as the optimization algorithm. The parameters are: learning rate=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08 $\endgroup$ – SunshineAtNoon Mar 3 '16 at 1:28

Based on my experience, just saying what i have seen, a few things could cause this: 1. learning rate. if it's too large, it could cause rising loss; just make it smaller, magnitudes smallers 2. in case of RNN, it could all blow up, weights, gradients, loss; there's paper published dealing with that.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.