step size in the first epochs of adam are too large [closed]

Question

When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below.

This looks like the step size (learning rate) is too large in the first step, but could be chosen larger in the last steps.

Is there an updated version of adam which does adjust the learning rate of the first epochs automatically? I think this behavior (too large first steps) is very typical for adam

Is there some literature that analyzes this phenomena?

_{PS: As far as I understand the Wikipedia article, it appears to me that the learning rate $\eta$ is fixed during adam?}

_{PPS: My intuition would tell me, that at the start you don't have an estimate for the momentum, so one should make short careful steps. And after some steps, when the estimate of the momentum improves, one could dare to make larger steps?}

Answer 1

you can experiment with https://keras.io/callbacks/#learningratescheduler to find propper setting, general idea is that at beginning you are probably in area that cause your NN to perform badly and area with "optimal" values have to be far away from the start so you should make big jumps and then you can decreas LR in order to gradient tune it self gently