45
$\begingroup$

I am training a model (Recurrent Neural Network) to classify 4 types of sequences. As I run my training I see the training loss going down until the point where I correctly classify over 90% of the samples in my training batches. However a couple of epochs later I notice that the training loss increases and that my accuracy drops. This seems weird to me as I would expect that on the training set the performance should improve with time not deteriorate. I am using cross entropy loss and my learning rate is 0.0002.

Update: It turned out that the learning rate was too high. With low a low enough learning rate I dont observe this behaviour. However I still find this peculiar. Any good explanations are welcome as to why this happens

Loss decreases and then increases

Improve this question
$\endgroup$
1
  • $\begingroup$ What's your code? $\endgroup$
    – High GPA
    Commented Aug 4, 2022 at 17:14

3 Answers 3

Reset to default
19
$\begingroup$

I had such a similar behavior when training a CNN, it was because I used the gradient descent with decaying learning rate for the error calculation. Have you significantly increased the number of iterations and checked if this behavior comes much later with the new low learning rate?

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ Actually yes however there comes a point that for a low enough learning rate where it will never go up again. I am looking for a theoretically sound explanation as to why this happens $\endgroup$
    – dins2018
    Commented Jan 26, 2018 at 21:46
  • $\begingroup$ Which optimization algorithm do you use? $\endgroup$
    – Freundlicher
    Commented Jan 26, 2018 at 21:55
  • 3
    $\begingroup$ I think you are approximate with this small learning rate so slowly to the local minimum that the point where the loss value slightly increases again (because you exceed the minimum) requires too many iterations. This increase in loss value is due to Adam, the moment the local minimum is exceeded and a certain number of iterations, a small number is divided by an even smaller number and the loss value explodes. $\endgroup$
    – Freundlicher
    Commented Jan 26, 2018 at 22:38
  • 3
    $\begingroup$ You can set beta1=0.9 and beta2=0.999. That are the common values that must work against this behavior. $\endgroup$
    – Freundlicher
    Commented Jan 26, 2018 at 22:48
  • 1
    $\begingroup$ I don't think the problem was learning rate. I just solved a problem where the training loss graph looked strikingly similar to this. In my case, I was also able to train for longer with a lower learning rate, but not to the same quality of results. I found out the issue was passing an invalid probability distribution to softmax cross entropy as labels (invalid meaning sum(prop(cl)) > 1. I had a bug where this happened in a small percentage of cases only. Although I haven't found this documented anywhere online, it seems to cause this kind of numerical instability. $\endgroup$
    – pseudopeach
    Commented Sep 10, 2020 at 18:07
21
$\begingroup$

With higher learning rates you are moving too much in the direction opposite to the gradient and may move away from the local minima which can increase the loss. Learning rate scheduling and gradient clipping can help.

Improve this answer
$\endgroup$
1
5
$\begingroup$

Because as learning rate is too big, it will diverge and fail to find the minimum of the loss function. Using a scheduler to decrease learning rate after certain epochs will help solve the problem

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.