I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below.

loss over time

The task itself if just trying to regress time series data with the squared difference loss. I half the learning rate every 3000 iterations, but the above is over ~2000 so the learning rate is constant. The batch size is 16 (GPU limitation), and initial learning rate is 0.0005.

What could the cause of these "two stages" be? I have seen this loss over time in papers before, but there usually isn't an explanation. Could it be getting out of a saddle point, or an artifact of the optimizer?


The two regimes of behavior probably reflect a saddle point, or a region of the parameter space that is just very shallow. This image gives a nice conceptual illustration.

enter image description here

Adaptive learning rates and momentum allow the optimizer to escape a region with a shallow gradient or saddle point. Quickly adapting to changing curvature is the goal of methods like , but some recent research has found that the gains due to Adam are marginal ("The Marginal Value of Adaptive Gradient Methods in Machine Learning" by Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht).

This can also happen if you’re reducing the learning rate at the particular epoch where the drop happens. But since you don’t mention that you’re specifically doing this, it’s probably due to a saddle point.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.