6
$\begingroup$

What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization?

$\endgroup$
  • 4
    $\begingroup$ For reasons that I don't understand, I've had conversations with people who conflate Adam and ADMM (alternating direction method of multipliers). Just an FYI for anyone passing through: these are not the same. $\endgroup$ – Sycorax Apr 18 '18 at 15:40
  • 2
    $\begingroup$ @Sycorax Don't get me started. If you think ML means machine learning you are in machine learning. If you think it means maximum likelihood you are firmly on the statistical side. If you have no idea either way, relax: many people enjoy life without knowing very much statistics. $\endgroup$ – Nick Cox Apr 18 '18 at 17:50
  • $\begingroup$ @NickCox Does familiarity with both imply that one is a CV moderator? $\endgroup$ – Sycorax Apr 18 '18 at 18:01
  • $\begingroup$ @Sycorax Knowing both meanings is clearly possible, as my comment would not have emerged otherwise! $\endgroup$ – Nick Cox Apr 18 '18 at 18:04
17
$\begingroup$

On p.1 of the document you cite: "the name Adam is derived from adaptive moment estimation".

$\endgroup$
  • $\begingroup$ Maybe the originator of the name could shine more light on this issue (e.g. maybe it is also the name of a professor, has to do with ada, or refers as well to amsterdam). @schaul $\endgroup$ – Martijn Weterings Apr 25 '18 at 9:50
  • $\begingroup$ As I understand it the originators are explaining the derivation in their paper introducing the method, so what other information is needed? In principle there might be a situation as with Bross's term ridit which decades later he explained as based on his wife's name; the justification he had given in public was just tongue-in-cheek. $\endgroup$ – Nick Cox Apr 25 '18 at 9:55
  • $\begingroup$ the question asks 'what is the history....' $\endgroup$ – Martijn Weterings Apr 25 '18 at 10:01
  • $\begingroup$ Sure. There is always more that could be said, e.g. precisely why that name, did we think of others and reject them, did someone else suggest it, etc. Not many of us invent new methods, but many of us have the same issue on a tiny scale with software we write. Even then it's often hard to remember about your small intellectual children and why they have those names. $\endgroup$ – Nick Cox Apr 25 '18 at 10:52
  • $\begingroup$ I was just pinging the inventor of the name (according to the linked article), and who knows he has something interesting to add. $\endgroup$ – Martijn Weterings Apr 25 '18 at 11:15
3
$\begingroup$

Although Nick already answered the question, I would like to elaborate a bit.

From the introduction of Adam: A Method for Stochastic Optimization (the original Adam paper):

The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients; the name Adam is derived from adaptive moment estimation.

Adam uses: $$\theta_{t+1}=\theta_{t}-\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$$ while:

  • $\theta_k$ is the vector of weights and bias in step $k$.
  • All of the operations are element-wise.
  • ${\hat m}_t$ is a bias-corrected moving average (implemented as an exponentially decaying average) of the gradients that were calculated until step $t$. In other words, $\hat m_t$ is an adaptive estimation of the first raw moment (i.e. the mean) of the gradient.
    Why "adaptive"? Because it is a weighted mean that gives more weight to gradients calculated closer to the current step, and gives virtually $0$ weight to gradients that were calculated in the distant past. In each step it is adapting to better estimate the mean of the gradient in the neighborhood of our current location in the cost function.
    (When I think about a moving average, I like to visualize a comet's trail, which becomes dimmer and dimmer as it gets further from the comet:
    a comet's trail as an intuition for a moving average)
  • Similarly, ${\hat v}_t$ is a bias-corrected moving average of the squares of the gradients. I.e. ${\hat v}_t$ is an adaptive estimation of the second raw moment (i.e. the uncentered variance) of the gradient.
  • $\alpha$ is a scalar that the paper refers to as "stepsize" and sometimes "learning rate".
  • Confusingly, the paper refers to $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$ as "stepsize" or "effective step in parameter space".

Thus, if I understand correctly, "learning rates" in the quote above refers to the components of $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$, and Adam is named after the computing of these components, which is mainly according to $\hat{m}_{t}$ and $\hat{v}_{t}$, the adaptive moment estimations.

It should also be noted that "adaptive" sometimes refers to using different learning rates for different parameters, in contrast to using the same learning rate for all parameters (sometimes called "global learning rate"). E.g. the basic stochastic gradient descent (SGD) uses $\theta_{t+1}=\theta_{t}-\eta g_{t}$, while $\eta$ is a scalar.
So if we think of $\hat{m}_{t}$ as parallel to $g_t$ in SGD, then Adam also has "adaptive" learning rates in this sense. (Though my guess is that this wasn't what the authors meant.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.