Etymology of "Adam" algorithm for gradient descent

Question

What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization?

Answer 1

On p.1 of the document you cite: "the name Adam is derived from adaptive moment estimation".

Answer 2

Although Nick already answered the question, I would like to elaborate a bit.

From the introduction of Adam: A Method for Stochastic Optimization (the original Adam paper):

The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients; the name Adam is derived from adaptive moment estimation.

Adam uses: $$\theta_{t+1}=\theta_{t}-\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$$ while:

• $\theta_k$ is the vector of weights and bias in step $k$.
• All of the operations are element-wise.
• ${\hat m}_t$ is a bias-corrected moving average (implemented as an exponentially decaying average) of the gradients that were calculated until step $t$. In other words, $\hat m_t$ is an adaptive estimation of the first raw moment (i.e. the mean) of the gradient.
  Why "adaptive"? Because it is a weighted mean that gives more weight to gradients calculated closer to the current step, and gives virtually $0$ weight to gradients that were calculated in the distant past. In each step it is adapting to better estimate the mean of the gradient in the neighborhood of our current location in the cost function.
  (When I think about a moving average, I like to visualize a comet's trail, which becomes dimmer and dimmer as it gets further from the comet:
  )
• Similarly, ${\hat v}_t$ is a bias-corrected moving average of the squares of the gradients. I.e. ${\hat v}_t$ is an adaptive estimation of the second raw moment (i.e. the uncentered variance) of the gradient.
• $\alpha$ is a scalar that the paper refers to as "stepsize" and sometimes "learning rate".
• Confusingly, the paper refers to $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$ as "stepsize" or "effective step in parameter space".

Thus, if I understand correctly, "learning rates" in the quote above refers to the components of $\frac{\alpha}{\sqrt{\hat{v}_{t}}+\epsilon}\hat{m}_{t}$, and Adam is named after the computing of these components, which is mainly according to $\hat{m}_{t}$ and $\hat{v}_{t}$, the adaptive moment estimations.

It should also be noted that "adaptive" sometimes refers to using different learning rates for different parameters, in contrast to using the same learning rate for all parameters (sometimes called "global learning rate"). E.g. the basic stochastic gradient descent (SGD) uses $\theta_{t+1}=\theta_{t}-\eta g_{t}$, while $\eta$ is a scalar.
So if we think of $\hat{m}_{t}$ as parallel to $g_t$ in SGD, then Adam also has "adaptive" learning rates in this sense. (Though my guess is that this wasn't what the authors meant.)