2
$\begingroup$

In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate".

To my understanding an adaptive learning rate should automatically change the value of the step sizes during the iterations. However, according to the pseudo-code 8.7 (see picture below) the step size $\epsilon$ is a constant.

Thus, in what is ADAM an adaptive learning rate algorithm?

enter image description here

$\endgroup$
  • 1
    $\begingroup$ It is because $\theta \leftarrow \theta + \Delta \theta$. At the end of every iteration, $ \Delta \theta$ will be updated because here, we compute the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively ($\hat{s}$ and $\hat{r}$). In order to decide our learning step, we multiply our learning rate by average of the gradient (as was the case with momentum) and divide it by the root mean square of the exponential average of square of gradients (as was the case with momentum) in equation of $\Delta \theta$. Then, we add the update. $\endgroup$ – ARAT Jan 10 '19 at 21:52
2
$\begingroup$

In the weight update, the contribution of bias correction of moments varies exponentially over epochs completed. Although the step_size/lr hyperparamter is constant, the contribution of gradients to updated weight varies over epochs,hence ADAPTIVE!!!!!!

$\endgroup$
  • $\begingroup$ thanks, i will set your new answer as the best one because I find it more intuitive to understand even if Robins says quite the same $\endgroup$ – S12000 Aug 24 '19 at 18:47
4
$\begingroup$

I think it is because you can see $ \epsilon s/(\sqrt{r} + \delta) $ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order moment are decreased.

$\endgroup$
  • $\begingroup$ Thanks, for the moment I will just keep the post open until someone confirms. As you say "I think it is" it suppose that you are not totally sure about the answer. $\endgroup$ – S12000 Dec 21 '18 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.