Hot answers tagged

46 votes
Accepted

How does the Adam method of stochastic gradient descent work?

The Adam paper says, "...many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking ...
user avatar
  • 29.8k
44 votes
Accepted

Adam optimizer with exponential decay

Empirically speaking: definitely try it out, you may find some very useful training heuristics, in which case, please do share! Usually people use some kind of decay, for Adam it seems uncommon. Is ...
user avatar
  • 6,742
26 votes

Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?

The problem of NOT correcting the bias According to the paper In case of sparse gradients, for a reliable estimate of the second moment one needs to average over many gradients by chosing a small ...
user avatar
  • 14.1k
23 votes
Accepted

The reason of superiority of Limited-memory BFGS over ADAM solver

There are a lot of reasons that this could be the case. Off the top of my head I can think of one plausible cause, but without knowing more about the problem it is difficult to suggest that it is the ...
user avatar
  • 1,593
22 votes
Accepted

Explanation of Spikes in training loss vs. iterations with Adam Optimizer

The spikes are an unavoidable consequence of Mini-Batch Gradient Descent in Adam (batch_size=32). Some mini-batches have 'by chance' unlucky data for the ...
user avatar
  • 1,036
21 votes

Explanation of Spikes in training loss vs. iterations with Adam Optimizer

I've spent insane amount of time debugging exploding gradients and similar behaviour. Your answer will be dependent on loss function, data, architecture etc. There's hundreds of reasons. I'll name a ...
user avatar
  • 674
18 votes
Accepted

Etymology of "Adam" algorithm for gradient descent

On p.1 of the document you cite: "the name Adam is derived from adaptive moment estimation".
user avatar
  • 49.2k
14 votes

How to choose between SGD with Nesterov momentum and Adam?

In general, there aren't definitive results on one learning algorithm being "better" than another. The common wisdom (which needs to be taken with a pound of salt) has been that Adam requires less ...
user avatar
13 votes

Adam optimizer with exponential decay

Adam uses the initial learning rate, or step size according to the original paper's terminology, while adaptively computing updates. Step size also gives an approximate bound for updates. In this ...
user avatar
13 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different ...
user avatar
  • 18.1k
13 votes
Accepted

Training a neural network on chess data

I think you need to consider running it on a GPU. Google Colab is free and Amazon AWS is very cheap. You seem to know what you are doing so you can probably get up and running with PyTorch very ...
user avatar
  • 49.8k
12 votes

RMSProp and Adam vs SGD

After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/...
user avatar
11 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning ...
user avatar
  • 4,329
11 votes
Accepted

What is a random variable in ADAM optimizer?

Converting my comment into an answer. The sentence right below your screenshot in the paper is the answer. The stochasticity might come from the evaluation at random subsamples (minibatches) of ...
user avatar
10 votes
Accepted

Is manually tuning learning rate during training redundant with optimization methods like Adam?

Yes, it is good practice to tune the learning rate even with Adam. Most variants of SGD which claim to be "adaptive", including optimizers like Adagrad and Adam, adjust the relative learning rates of ...
user avatar
  • 23.2k
10 votes

What does Diagonal Rescaling of the gradients mean in ADAM paper?

The original Adam paper briefly explains what it means by "invariant to diagonal rescaling of the gradients" at the end of section 2.1. I would try to explain it in some more detail. Like ...
user avatar
  • 1,182
9 votes

Adam optimizer with exponential decay

The reason why most people don't use learning rate decay with Adam is that the algorithm itself does a learning rate decay in the following way: ...
user avatar
  • 295
9 votes

How does batch size affect Adam Optimizer?

Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, ...
user avatar
8 votes
Accepted

What should we do when changing SGD optimizer to Adam optimizer?

In my experience, changing optimizers is not a simple matter of swapping one for the other. Instead, changing optimizers also interacts with several other configuration choices in the neural network. ...
user avatar
  • 79.6k
8 votes

No change in accuracy using Adam Optimizer when SGD works fine

The benefits of Adam can be marginal, at best. The initial results were strong, but there is evidence that Adam converges to dramatically different minima compared to SGD (or SGD + momentum). "The ...
user avatar
  • 79.6k
5 votes

How does batch size affect Adam Optimizer?

I would just leave this as a comment, but I don't have enough reputation. There's an excellent discussion of the trade offs of large and small batch sizes here.
user avatar
  • 76
5 votes

What does decay_steps mean in Tensorflow tf.train.exponential_decay?

As mentioned in the code of the function the relation of decay_steps with decayed_learning_rate is the following: ...
user avatar
  • 1,037
4 votes
Accepted

Why update all parameters at each step of the Adam optimiser even when we have sparse observations?

EDIT: Apparently tensorflow has a "Lazy Adam Optimizer" that only updates the gradient for variables whose indices appear in the current batch. Lazy_Adam_Optimizer This may be a good idea for very ...
user avatar
4 votes
Accepted

Two large decreses in loss function with ADAM optimizer

The two regimes of behavior probably reflect a saddle point, or a region of the parameter space that is just very shallow. This image gives a nice conceptual illustration. Adaptive learning rates ...
user avatar
  • 79.6k
4 votes
Accepted

How to select parameters for ADAM gradient descent

$\beta_1$ and $\beta_2$ are are the forgetting parameters for the Adam optimizer. The lower the either one, the faster the running average is updated (and hence the faster previous gradients are ...
user avatar
4 votes

Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant

I think it is because you can see $ \epsilon s/(\sqrt{r} + \delta) $ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order ...
user avatar
4 votes

How well should I expect Adam to work?

There might be several factors at play here: The optimal learning rate for momentum based algorithms is usually lower than that for plain GD, because momentum increases the effective step size. I ...
user avatar
  • 23.2k
4 votes

When to use weight decay for ADAM optimiser?

Weight decay is a form of regularization that changes the objective function. You can also use other regularization techniques if you’d like. Either way, weight decay does alter the values used to ...
user avatar
3 votes

Etymology of "Adam" algorithm for gradient descent

Although Nick already answered the question, I would like to elaborate a bit. From the introduction of Adam: A Method for Stochastic Optimization (the original Adam paper): The method computes ...
user avatar
  • 1,182
3 votes

Adam optimizer with exponential decay

I agree with @Indie AI's opinion, here I supply some other information: From CS231n: ... Many of these methods may still require other hyperparameter settings, but the argument is that they are ...
user avatar
  • 131

Only top scored, non community-wiki answers of a minimum length are eligible