I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below.
The task itself if just trying to regress time series data with the squared difference loss. I half the learning rate every 3000 iterations, but the above is over ~2000 so the learning rate is constant. The batch size is 16 (GPU limitation), and initial learning rate is 0.0005.
What could the cause of these "two stages" be? I have seen this loss over time in papers before, but there usually isn't an explanation. Could it be getting out of a saddle point, or an artifact of the optimizer?