What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

Question

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville:

Adam is generally regarded as being fairly robust to the choice of hyper parameters, though the learning rate sometimes needs to be changed from the suggested default.

if this is true its a big deal because hyper parameter search can be really important (in my experience at least) in the statistical performance of a deep learning system. Thus, my question is, why is Adam Robust to such important parameters? Specially $\beta_1$ and $\beta_2$?

I've read the Adam paper and it doesn't provide any explanation to why it works with those parameters or why its robust. Do they justify that elsewhere?

Also, as I read the paper, it seems that the number of hyper parameters they tried where very small, for $\beta_1$ only 2 and for $\beta_2$ only 3. How can this be a thorough empirical study if it only works on 2x3 hyper parameters?

Answer 1

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different values for $\beta_1$, $\beta_2$ and $\alpha$.

Personally, I don't find their argument convincing, in particular because they do not present results across a variety of problems. With that said, I will note that I have used ADAM for a variety of problems, and my personal finding is that the default values of $\beta_1$ and $\beta_2$ do seem surprisingly reliable, although a good deal of fiddling with $\alpha$ is required.

Answer 2

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning rates decay. If they decay really fast, then the learning rates will jump about all over the place. If they decay slowly, it will take ages for the learning rates to be learned. But note that in all cases, the learning rates are determined automatically, based on a moving estimate of the per-parameter gradient, and the per-parameter squared gradient.

This is in huge contrast with stock vanilla Stochastic Gradient Descent, where:

• learning rates are not per-parameter, but there is a single, global learning rate, that is applied bluntly, across all parameters
  • (by the way, this is one reason why data is often whitened, normalized, prior to being sent into nets, to try to keep the ideal per-parameter weights similar-ish)
• the learning rate provided is the exact learning rate used, and wont adapt over time

Adam is not the only optimizer with adaptive learning rates. As the Adam paper states itself, it's highly related to Adagrad and Rmsprop, which are also extremely insensitive to hyperparameters. Especially, Rmsprop works quite nicely.

But Adam is the best in general. With very few exceptions Adam will do what you want :)

There are a few fairly pathological cases where Adam will not work, particularly for some very non-stationary distributions. In these cases, Rmsprop is an excellent standby option. But generally speaking, for most non-pathological cases, Adam works extremely well.