When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below.
This looks like the step size (learning rate) is too large in the first step, but could be chosen larger in the last steps.
Is there an updated version of adam which does adjust the learning rate of the first epochs automatically? I think this behavior (too large first steps) is very typical for adam
Is there some literature that analyzes this phenomena?
PS: As far as I understand the Wikipedia article, it appears to me that the learning rate $\eta$ is fixed during adam?
PPS: My intuition would tell me, that at the start you don't have an estimate for the momentum, so one should make short careful steps. And after some steps, when the estimate of the momentum improves, one could dare to make larger steps?