24
$\begingroup$

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville:

Adam is generally regarded as being fairly robust to the choice of hyper parameters, though the learning rate sometimes needs to be changed from the suggested default.

if this is true its a big deal because hyper parameter search can be really important (in my experience at least) in the statistical performance of a deep learning system. Thus, my question is, why is Adam Robust to such important parameters? Specially $\beta_1$ and $\beta_2$?

I've read the Adam paper and it doesn't provide any explanation to why it works with those parameters or why its robust. Do they justify that elsewhere?

Also, as I read the paper, it seems that the number of hyper parameters they tried where very small, for $\beta_1$ only 2 and for $\beta_2$ only 3. How can this be a thorough empirical study if it only works on 2x3 hyper parameters?

$\endgroup$
  • 1
    $\begingroup$ Send an email to the authors of the book who made the claim. Ask them what the claim is based on. $\endgroup$ – Mark L. Stone Sep 14 '16 at 23:22
  • 1
    $\begingroup$ @MarkL.Stone he only said he said it because the abstract said it. Hardly a convincing argument. Maybe next time I will e-mail the authors of the actual paper. $\endgroup$ – Charlie Parker Sep 20 '16 at 1:18
  • 7
    $\begingroup$ And so self-promotion becomes fact. $\endgroup$ – Mark L. Stone Sep 20 '16 at 1:46
  • $\begingroup$ @MarkL.Stone in his defense, he might have been to busy to answer properly and I only contacted 1 of the 3 authors. Maybe I could contact the others but I'm not sure if they will answer given (at least) one is a professor. With the hype in DL I bet he gets 300 e-mails daily. $\endgroup$ – Charlie Parker Sep 20 '16 at 16:59
  • 7
    $\begingroup$ Now that the book is out, the Adam authors have confirmation of how great their algorithm is. Reminds me of '89 Bay Area earthquake. News radio station made unconfirmed report of # of fatalities on highway collapse - said they were seeking confirmation from governor's office. Then they got the governor on the phone and asked if he could confirm # of fatalities. He said that's what he heard. The radio station then reported that they now had confirmation from the governor. It turns out that the governor meant that he heard it, as it turns out, on that radio station. So, circular confirmation. $\endgroup$ – Mark L. Stone Sep 20 '16 at 23:35
7
$\begingroup$

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different values for $\beta_1$, $\beta_2$ and $\alpha$.

Personally, I don't find their argument convincing, in particular because they do not present results across a variety of problems. With that said, I will note that I have used ADAM for a variety of problems, and my personal finding is that the default values of $\beta_1$ and $\beta_2$ do seem surprisingly reliable, although a good deal of fiddling with $\alpha$ is required.

$\endgroup$
5
$\begingroup$

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning rates decay. If they decay really fast, then the learning rates will jump about all over the place. If they decay slowly, it will take ages for the learning rates to be learned. But note that in all cases, the learning rates are determined automatically, based on a moving estimate of the per-parameter gradient, and the per-parameter squared gradient.

This is in huge contrast with stock vanilla Stochastic Gradient Descent, where:

  • learning rates are not per-parameter, but there is a single, global learning rate, that is applied bluntly, across all parameters
    • (by the way, this is one reason why data is often whitened, normalized, prior to being sent into nets, to try to keep the ideal per-parameter weights similar-ish)
  • the learning rate provided is the exact learning rate used, and wont adapt over time

Adam is not the only optimizer with adaptive learning rates. As the Adam paper states itself, it's highly related to Adagrad and Rmsprop, which are also extremely insensitive to hyperparameters. Especially, Rmsprop works quite nicely.

But Adam is the best in general. With very few exceptions Adam will do what you want :)

There are a few fairly pathological cases where Adam will not work, particularly for some very non-stationary distributions. In these cases, Rmsprop is an excellent standby option. But generally speaking, for most non-pathological cases, Adam works extremely well.

$\endgroup$
  • 1
    $\begingroup$ I don't get it, why is Adam robust to values of $\beta_1,\beta_2$. There seems to be no study whatsoever to back it up. Seems folks "wisdom" as far as I can tell. $\endgroup$ – Charlie Parker Oct 10 '17 at 3:42
  • $\begingroup$ Yes, if you mean, 'is there an opportunity here to research deeper into why?', well... maybe. $\endgroup$ – Hugh Perkins Oct 10 '17 at 8:56
  • 2
    $\begingroup$ this is not a "deeper question". This is it seems one of the most important point of the paper, no? The whole point is that it does things by "itself" but then there are these other hyperparameters that seems to be magically robust. Thats the issue I have. Its seems to me to be related to the core of the paper unless I got misunderstood the point of Adam. $\endgroup$ – Charlie Parker Oct 10 '17 at 16:10
  • $\begingroup$ "There are a few fairly pathological cases where Adam will not work, particularly for some very non-stationary distributions." <- any references here? $\endgroup$ – mimoralea Oct 6 '18 at 18:03
0
$\begingroup$

Looking at the ADAM formulas, it seems a bit puzzling that after a very large number of batch iterations (say ~400k) the magnitude of the, original error based, gradient itself plays ~no actual role in the taken step which seems to aspire to the learning rate configuration parameter in the relevant sign.
Perhaps ADAM controls the weight adaptation better than simple SGD during first iterations/epocs but going forward the update seems to be reduced to something somewhat naive (?) Can anyone provide some intuition on why that is actually desired and/or tends to work well ?

$\endgroup$
  • $\begingroup$ Actually seems like the error based gradient magnitude itself doesn't play a real role even form the start. Question is why is such normalization working well, and what does it imply regarding the GD intuition guiding DL and other common learning models ? $\endgroup$ – Danny Rosen May 19 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.