46
votes
Accepted
How does the Adam method of stochastic gradient descent work?
The Adam paper says, "...many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking ...
- 29.8k
44
votes
Accepted
Adam optimizer with exponential decay
Empirically speaking: definitely try it out, you may find some very useful training heuristics, in which case, please do share!
Usually people use some kind of decay, for Adam it seems uncommon. Is ...
- 6,742
26
votes
Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?
The problem of NOT correcting the bias
According to the paper
In case of sparse gradients, for a reliable estimate of the second moment one needs to average over
many gradients by chosing a small ...
- 14.1k
23
votes
Accepted
The reason of superiority of Limited-memory BFGS over ADAM solver
There are a lot of reasons that this could be the case. Off the top of my head I can think of one plausible cause, but without knowing more about the problem it is difficult to suggest that it is the ...
- 1,593
22
votes
Accepted
Explanation of Spikes in training loss vs. iterations with Adam Optimizer
The spikes are an unavoidable consequence of Mini-Batch Gradient Descent in Adam (batch_size=32).
Some mini-batches have 'by chance' unlucky data for the ...
- 1,036
21
votes
Explanation of Spikes in training loss vs. iterations with Adam Optimizer
I've spent insane amount of time debugging exploding gradients and similar behaviour. Your answer will be dependent on loss function, data, architecture etc. There's hundreds of reasons. I'll name a ...
- 674
18
votes
Accepted
Etymology of "Adam" algorithm for gradient descent
On p.1 of the document you cite: "the name Adam is derived from adaptive moment estimation".
- 49.2k
14
votes
How to choose between SGD with Nesterov momentum and Adam?
In general, there aren't definitive results on one learning algorithm being "better" than another. The common wisdom (which needs to be taken with a pound of salt) has been that Adam requires less ...
- 351
13
votes
Adam optimizer with exponential decay
Adam uses the initial learning rate, or step size according to the original paper's terminology, while adaptively computing updates. Step size also gives an approximate bound for updates. In this ...
13
votes
What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?
In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different ...
- 18.1k
13
votes
Accepted
Training a neural network on chess data
I think you need to consider running it on a GPU. Google Colab is free and Amazon AWS is very cheap. You seem to know what you are doing so you can probably get up and running with PyTorch very ...
- 49.8k
12
votes
RMSProp and Adam vs SGD
After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/...
11
votes
What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?
Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning ...
- 4,329
11
votes
Accepted
What is a random variable in ADAM optimizer?
Converting my comment into an answer.
The sentence right below your screenshot in the paper is the answer.
The stochasticity might come from the evaluation at random subsamples (minibatches) of ...
- 6,952
10
votes
Accepted
Is manually tuning learning rate during training redundant with optimization methods like Adam?
Yes, it is good practice to tune the learning rate even with Adam.
Most variants of SGD which claim to be "adaptive", including optimizers like Adagrad and Adam, adjust the relative learning rates of ...
- 23.2k
10
votes
What does Diagonal Rescaling of the gradients mean in ADAM paper?
The original Adam paper briefly explains what it means by "invariant to diagonal rescaling of the gradients" at the end of section 2.1.
I would try to explain it in some more detail.
Like ...
- 1,182
9
votes
Adam optimizer with exponential decay
The reason why most people don't use learning rate decay with Adam is that the algorithm itself does a learning rate decay in the following way:
...
- 295
9
votes
How does batch size affect Adam Optimizer?
Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, ...
- 209
8
votes
Accepted
What should we do when changing SGD optimizer to Adam optimizer?
In my experience, changing optimizers is not a simple matter of swapping one for the other. Instead, changing optimizers also interacts with several other configuration choices in the neural network.
...
- 79.6k
8
votes
No change in accuracy using Adam Optimizer when SGD works fine
The benefits of Adam can be marginal, at best. The initial results were strong, but there is evidence that Adam converges to dramatically different minima compared to SGD (or SGD + momentum).
"The ...
- 79.6k
5
votes
How does batch size affect Adam Optimizer?
I would just leave this as a comment, but I don't have enough reputation.
There's an excellent discussion of the trade offs of large and small batch sizes here.
- 76
5
votes
What does decay_steps mean in Tensorflow tf.train.exponential_decay?
As mentioned in the code of the function the relation of decay_steps with decayed_learning_rate is the following:
...
- 1,037
4
votes
Accepted
Why update all parameters at each step of the Adam optimiser even when we have sparse observations?
EDIT:
Apparently tensorflow has a "Lazy Adam Optimizer" that only updates the gradient for variables whose indices appear in the current batch.
Lazy_Adam_Optimizer
This may be a good idea for very ...
4
votes
Accepted
Two large decreses in loss function with ADAM optimizer
The two regimes of behavior probably reflect a saddle point, or a region of the parameter space that is just very shallow. This image gives a nice conceptual illustration.
Adaptive learning rates ...
- 79.6k
4
votes
Accepted
How to select parameters for ADAM gradient descent
$\beta_1$ and $\beta_2$ are are the forgetting parameters for the Adam optimizer. The lower the either one, the faster the running average is updated (and hence the faster previous gradients are ...
- 156
4
votes
Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant
I think it is because you can see $ \epsilon s/(\sqrt{r} + \delta) $ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order ...
- 184
4
votes
How well should I expect Adam to work?
There might be several factors at play here:
The optimal learning rate for momentum based algorithms is usually lower than that for plain GD, because momentum increases the effective step size. I ...
- 23.2k
4
votes
When to use weight decay for ADAM optimiser?
Weight decay is a form of regularization that changes the objective function. You can also use other regularization techniques if you’d like.
Either way, weight decay does alter the values used to ...
- 6,952
3
votes
Etymology of "Adam" algorithm for gradient descent
Although Nick already answered the question, I would like to elaborate a bit.
From the introduction of Adam: A Method for Stochastic Optimization (the original Adam paper):
The method computes ...
- 1,182
3
votes
Adam optimizer with exponential decay
I agree with @Indie AI's opinion, here I supply some other information:
From CS231n:
... Many of these methods may still require other hyperparameter settings, but the argument is that they are ...
- 131
Only top scored, non community-wiki answers of a minimum length are eligible