2
$\begingroup$

As you can see in the picture, when the loss reaches 1.54, it goes up and then drops to the same number again and again. But if I reduce the learning rate, the loss is maintained around 1.6, and goes neither up nor down.

Training loss

$\endgroup$
7
  • 1
    $\begingroup$ We are talking about a 0.2 difference between min and max. Is this significant for your problem? $\endgroup$ Commented Mar 26, 2020 at 11:20
  • $\begingroup$ I want the loss keep going down to zero, rather than jump up and down. $\endgroup$
    – YINQ
    Commented Mar 26, 2020 at 11:25
  • 1
    $\begingroup$ You already figured out that if you change the learning rate the loss stabilizes, so that may be a better choice for your problem. And it looks like the loss won't go much further down than this. Also your loss is lowest right at the start, implying that you may be just wasting your time using more iterations because you are not gaining anything. $\endgroup$ Commented Mar 26, 2020 at 11:28
  • $\begingroup$ Is there no other possibility here?The weight of the parameter is still changing largely in every epoch. $\endgroup$
    – YINQ
    Commented Mar 26, 2020 at 11:38
  • $\begingroup$ What is "the weight of the parameter"? Which parameter? How is it weighted? Why is changing a large amount important? $\endgroup$
    – Sycorax
    Commented Mar 26, 2020 at 13:01

1 Answer 1

2
$\begingroup$

As it's also pointed out in the comments, it seems the network is not learning any more. The up and downs in the loss are probably because you're around a local minimum and the learning rate is relatively large for that neighbourhood so reducing it stabilises your loss.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.