Two large decreses in loss function with ADAM optimizer

Question

I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below.

The task itself if just trying to regress time series data with the squared difference loss. I half the learning rate every 3000 iterations, but the above is over ~2000 so the learning rate is constant. The batch size is 16 (GPU limitation), and initial learning rate is 0.0005.

What could the cause of these "two stages" be? I have seen this loss over time in papers before, but there usually isn't an explanation. Could it be getting out of a saddle point, or an artifact of the optimizer?

Answer 1

The two regimes of behavior probably reflect a saddle point, or a region of the parameter space that is just very shallow. This image gives a nice conceptual illustration.

Adaptive learning rates and momentum allow the optimizer to escape a region with a shallow gradient or saddle point. Quickly adapting to changing curvature is the goal of methods like adam, but some recent research has found that the gains due to Adam are marginal ("The Marginal Value of Adaptive Gradient Methods in Machine Learning" by Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht).

This can also happen if you’re reducing the learning rate at the particular epoch where the drop happens. But since you don’t mention that you’re specifically doing this, it’s probably due to a saddle point.