3
$\begingroup$

In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate".

To my understanding an adaptive learning rate should automatically change the value of the step sizes during the iterations. However, according to the pseudo-code 8.7 (see picture below) the step size $\epsilon$ is a constant.

Thus, in what is ADAM an adaptive learning rate algorithm?

enter image description here

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ It is because $\theta \leftarrow \theta + \Delta \theta$. At the end of every iteration, $ \Delta \theta$ will be updated because here, we compute the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively ($\hat{s}$ and $\hat{r}$). In order to decide our learning step, we multiply our learning rate by average of the gradient (as was the case with momentum) and divide it by the root mean square of the exponential average of square of gradients (as was the case with momentum) in equation of $\Delta \theta$. Then, we add the update. $\endgroup$
    – ARAT
    Jan 10, 2019 at 21:52

2 Answers 2

Reset to default
3
$\begingroup$

In the weight update, the contribution of bias correction of moments varies exponentially over epochs completed. Although the step_size/lr hyperparamter is constant, the contribution of gradients to updated weight varies over epochs,hence ADAPTIVE!!!!!!

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks, i will set your new answer as the best one because I find it more intuitive to understand even if Robins says quite the same $\endgroup$
    – S12000
    Aug 24, 2019 at 18:47
5
$\begingroup$

I think it is because you can see $ \epsilon s/(\sqrt{r} + \delta) $ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order moment are decreased.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, for the moment I will just keep the post open until someone confirms. As you say "I think it is" it suppose that you are not totally sure about the answer. $\endgroup$
    – S12000
    Dec 21, 2018 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.