# Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant

In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate".

To my understanding an adaptive learning rate should automatically change the value of the step sizes during the iterations. However, according to the pseudo-code 8.7 (see picture below) the step size $$\epsilon$$ is a constant. • It is because $\theta \leftarrow \theta + \Delta \theta$. At the end of every iteration, $\Delta \theta$ will be updated because here, we compute the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively ($\hat{s}$ and $\hat{r}$). In order to decide our learning step, we multiply our learning rate by average of the gradient (as was the case with momentum) and divide it by the root mean square of the exponential average of square of gradients (as was the case with momentum) in equation of $\Delta \theta$. Then, we add the update.
– ARAT
Jan 10, 2019 at 21:52

I think it is because you can see $$\epsilon s/(\sqrt{r} + \delta)$$ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order moment are decreased.