I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville:

Adam is generally regarded as being fairly robust to the choice of hyper parameters, though the learning rate sometimes needs to be changed from the suggested default.

if this is true its a big deal because hyper parameter search can be really important (in my experience at least) in the statistical performance of a deep learning system. Thus, my question is, why is Adam Robust to such important parameters? Specially $\beta_1$ and $\beta_2$?

I've read the Adam paper and it doesn't provide any explanation to why it works with those parameters or why its robust. Do they justify that elsewhere?

Also, as I read the paper, it seems that the number of hyper parameters they tried where very small, for $\beta_1$ only 2 and for $\beta_2$ only 3. How can this be a thorough empirical study if it only works on 2x3 hyper parameters?

  • 3
    $\begingroup$ Send an email to the authors of the book who made the claim. Ask them what the claim is based on. $\endgroup$ Commented Sep 14, 2016 at 23:22
  • 3
    $\begingroup$ @MarkL.Stone he only said he said it because the abstract said it. Hardly a convincing argument. Maybe next time I will e-mail the authors of the actual paper. $\endgroup$ Commented Sep 20, 2016 at 1:18
  • 8
    $\begingroup$ And so self-promotion becomes fact. $\endgroup$ Commented Sep 20, 2016 at 1:46
  • 1
    $\begingroup$ @MarkL.Stone in his defense, he might have been to busy to answer properly and I only contacted 1 of the 3 authors. Maybe I could contact the others but I'm not sure if they will answer given (at least) one is a professor. With the hype in DL I bet he gets 300 e-mails daily. $\endgroup$ Commented Sep 20, 2016 at 16:59
  • 10
    $\begingroup$ Now that the book is out, the Adam authors have confirmation of how great their algorithm is. Reminds me of '89 Bay Area earthquake. News radio station made unconfirmed report of # of fatalities on highway collapse - said they were seeking confirmation from governor's office. Then they got the governor on the phone and asked if he could confirm # of fatalities. He said that's what he heard. The radio station then reported that they now had confirmation from the governor. It turns out that the governor meant that he heard it, as it turns out, on that radio station. So, circular confirmation. $\endgroup$ Commented Sep 20, 2016 at 23:35

2 Answers 2


In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different values for $\beta_1$, $\beta_2$ and $\alpha$.

Personally, I don't find their argument convincing, in particular because they do not present results across a variety of problems. With that said, I will note that I have used ADAM for a variety of problems, and my personal finding is that the default values of $\beta_1$ and $\beta_2$ do seem surprisingly reliable, although a good deal of fiddling with $\alpha$ is required.


Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning rates decay. If they decay really fast, then the learning rates will jump about all over the place. If they decay slowly, it will take ages for the learning rates to be learned. But note that in all cases, the learning rates are determined automatically, based on a moving estimate of the per-parameter gradient, and the per-parameter squared gradient.

This is in huge contrast with stock vanilla Stochastic Gradient Descent, where:

  • learning rates are not per-parameter, but there is a single, global learning rate, that is applied bluntly, across all parameters
    • (by the way, this is one reason why data is often whitened, normalized, prior to being sent into nets, to try to keep the ideal per-parameter weights similar-ish)
  • the learning rate provided is the exact learning rate used, and wont adapt over time

Adam is not the only optimizer with adaptive learning rates. As the Adam paper states itself, it's highly related to Adagrad and Rmsprop, which are also extremely insensitive to hyperparameters. Especially, Rmsprop works quite nicely.

But Adam is the best in general. With very few exceptions Adam will do what you want :)

There are a few fairly pathological cases where Adam will not work, particularly for some very non-stationary distributions. In these cases, Rmsprop is an excellent standby option. But generally speaking, for most non-pathological cases, Adam works extremely well.

  • 2
    $\begingroup$ I don't get it, why is Adam robust to values of $\beta_1,\beta_2$. There seems to be no study whatsoever to back it up. Seems folks "wisdom" as far as I can tell. $\endgroup$ Commented Oct 10, 2017 at 3:42
  • 1
    $\begingroup$ Yes, if you mean, 'is there an opportunity here to research deeper into why?', well... maybe. $\endgroup$ Commented Oct 10, 2017 at 8:56
  • 3
    $\begingroup$ this is not a "deeper question". This is it seems one of the most important point of the paper, no? The whole point is that it does things by "itself" but then there are these other hyperparameters that seems to be magically robust. Thats the issue I have. Its seems to me to be related to the core of the paper unless I got misunderstood the point of Adam. $\endgroup$ Commented Oct 10, 2017 at 16:10
  • $\begingroup$ "There are a few fairly pathological cases where Adam will not work, particularly for some very non-stationary distributions." <- any references here? $\endgroup$
    – mimoralea
    Commented Oct 6, 2018 at 18:03
  • 1
    $\begingroup$ @HughPerkins what about their extensions: AdaMax, AdamW? $\endgroup$
    – Alex
    Commented Aug 7, 2020 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.