5
$\begingroup$

I'm training a simple feed-forward Deep Network with standard relu/sigmoid activation functions, using Tensorflow. I tried several optimizers and I noticed that those belonging to the Adam family show a strange behavior in loss trend, as you can see in the picture.

enter image description here

I've read about some similar issues, but their loss trends appear to be quite different than mine.

Have you any idea of the possible source of this problem? Can you suggest some possible references to investigate this issue?

However, I verified that using SGD or RMSprop as optimizer the problem disappeared, but the obtained final loss value was much higher.

Improve this question
$\endgroup$
3
  • $\begingroup$ Seems like it could be the same problem as in the other post that you linked. $\endgroup$
    – Aaron
    Commented Sep 12, 2017 at 22:52
  • $\begingroup$ @Aaron, yes the problem is similar, but in the other post the loss diverges, while in mine the trend is not changed by the spikes: their effects are only local. $\endgroup$
    – user38320
    Commented Sep 13, 2017 at 7:20
  • $\begingroup$ Possibly related: stats.stackexchange.com/questions/169216/… although the fact that this disappears when using SGD instead of Adam suggests that the difference is related to the optimizer. $\endgroup$
    – Sycorax ♦
    Commented Oct 24, 2018 at 13:42

0

Reset to default