2
$\begingroup$

I'm training a two-layer LSTM with Adam optimizer for time-series data. I encountered several times that there was a "turning point" of the MAE vs. epochs plot. Is this a normal behavior?

enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

I have sometimes observed it, sometimes even repeatedly, but by no means always, and also with other types of DNNs. I then assumed that the optimizer, after searching unsuccessfully on a plateau of the loss surface, finally found a local minimum which is then exploited. The layers usually get initialized via He or Glorot (Xavier), i.e. from a distribution with a relatively small standard deviation, and it seems to require some epochs to get to the more favorable weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.