I'm training a two-layer LSTM with Adam optimizer for time-series data. I encountered several times that there was a "turning point" of the MAE vs. epochs plot. Is this a normal behavior?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I have sometimes observed it, sometimes even repeatedly, but by no means always, and also with other types of DNNs. I then assumed that the optimizer, after searching unsuccessfully on a plateau of the loss surface, finally found a local minimum which is then exploited. The layers usually get initialized via He or Glorot (Xavier), i.e. from a distribution with a relatively small standard deviation, and it seems to require some epochs to get to the more favorable weights.
