As you can see in the picture, when the loss reaches 1.54, it goes up and then drops to the same number again and again. But if I reduce the learning rate, the loss is maintained around 1.6, and goes neither up nor down.
$\begingroup$
$\endgroup$
7
-
1$\begingroup$ We are talking about a 0.2 difference between min and max. Is this significant for your problem? $\endgroup$– user2974951Mar 26, 2020 at 11:20
-
$\begingroup$ I want the loss keep going down to zero, rather than jump up and down. $\endgroup$– YINQMar 26, 2020 at 11:25
-
1$\begingroup$ You already figured out that if you change the learning rate the loss stabilizes, so that may be a better choice for your problem. And it looks like the loss won't go much further down than this. Also your loss is lowest right at the start, implying that you may be just wasting your time using more iterations because you are not gaining anything. $\endgroup$– user2974951Mar 26, 2020 at 11:28
-
$\begingroup$ Is there no other possibility here？The weight of the parameter is still changing largely in every epoch. $\endgroup$– YINQMar 26, 2020 at 11:38
-
$\begingroup$ What is "the weight of the parameter"? Which parameter? How is it weighted? Why is changing a large amount important? $\endgroup$– Sycorax ♦Mar 26, 2020 at 13:01
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
As it's also pointed out in the comments, it seems the network is not learning any more. The up and downs in the loss are probably because you're around a local minimum and the learning rate is relatively large for that neighbourhood so reducing it stabilises your loss.
