# Tag Info

Accepted

The Adam paper says, "...many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking ...
• 29.8k
Accepted

### Adam optimizer with exponential decay

Empirically speaking: definitely try it out, you may find some very useful training heuristics, in which case, please do share! Usually people use some kind of decay, for Adam it seems uncommon. Is ...
• 6,742

### Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?

The problem of NOT correcting the bias According to the paper In case of sparse gradients, for a reliable estimate of the second moment one needs to average over many gradients by chosing a small ...
• 14.1k
Accepted

### The reason of superiority of Limited-memory BFGS over ADAM solver

There are a lot of reasons that this could be the case. Off the top of my head I can think of one plausible cause, but without knowing more about the problem it is difficult to suggest that it is the ...
• 1,593
Accepted

### Explanation of Spikes in training loss vs. iterations with Adam Optimizer

The spikes are an unavoidable consequence of Mini-Batch Gradient Descent in Adam (batch_size=32). Some mini-batches have 'by chance' unlucky data for the ...
• 1,036

### Explanation of Spikes in training loss vs. iterations with Adam Optimizer

I've spent insane amount of time debugging exploding gradients and similar behaviour. Your answer will be dependent on loss function, data, architecture etc. There's hundreds of reasons. I'll name a ...
• 674
Accepted

On p.1 of the document you cite: "the name Adam is derived from adaptive moment estimation".
• 49.2k

### How to choose between SGD with Nesterov momentum and Adam?

In general, there aren't definitive results on one learning algorithm being "better" than another. The common wisdom (which needs to be taken with a pound of salt) has been that Adam requires less ...
• 351

### Adam optimizer with exponential decay

Adam uses the initial learning rate, or step size according to the original paper's terminology, while adaptively computing updates. Step size also gives an approximate bound for updates. In this ...

### What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different ...
• 18.1k
Accepted

### Training a neural network on chess data

I think you need to consider running it on a GPU. Google Colab is free and Amazon AWS is very cheap. You seem to know what you are doing so you can probably get up and running with PyTorch very ...
• 49.8k

### RMSProp and Adam vs SGD

After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/...

### What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning ...
• 4,329
Accepted

### What is a random variable in ADAM optimizer?

Converting my comment into an answer. The sentence right below your screenshot in the paper is the answer. The stochasticity might come from the evaluation at random subsamples (minibatches) of ...
• 6,952
Accepted

### Is manually tuning learning rate during training redundant with optimization methods like Adam?

Yes, it is good practice to tune the learning rate even with Adam. Most variants of SGD which claim to be "adaptive", including optimizers like Adagrad and Adam, adjust the relative learning rates of ...
• 23.2k

### What does Diagonal Rescaling of the gradients mean in ADAM paper?

The original Adam paper briefly explains what it means by "invariant to diagonal rescaling of the gradients" at the end of section 2.1. I would try to explain it in some more detail. Like ...
• 1,182

### Adam optimizer with exponential decay

The reason why most people don't use learning rate decay with Adam is that the algorithm itself does a learning rate decay in the following way: ...
• 295

### How does batch size affect Adam Optimizer?

Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, ...
• 209
Accepted

### What should we do when changing SGD optimizer to Adam optimizer?

In my experience, changing optimizers is not a simple matter of swapping one for the other. Instead, changing optimizers also interacts with several other configuration choices in the neural network. ...
• 79.6k

### No change in accuracy using Adam Optimizer when SGD works fine

The benefits of Adam can be marginal, at best. The initial results were strong, but there is evidence that Adam converges to dramatically different minima compared to SGD (or SGD + momentum). "The ...
• 79.6k

### How does batch size affect Adam Optimizer?

I would just leave this as a comment, but I don't have enough reputation. There's an excellent discussion of the trade offs of large and small batch sizes here.
• 76

### What does decay_steps mean in Tensorflow tf.train.exponential_decay?

As mentioned in the code of the function the relation of decay_steps with decayed_learning_rate is the following: ...
• 1,037
Accepted

### Why update all parameters at each step of the Adam optimiser even when we have sparse observations?

EDIT: Apparently tensorflow has a "Lazy Adam Optimizer" that only updates the gradient for variables whose indices appear in the current batch. Lazy_Adam_Optimizer This may be a good idea for very ...
Accepted

### Two large decreses in loss function with ADAM optimizer

The two regimes of behavior probably reflect a saddle point, or a region of the parameter space that is just very shallow. This image gives a nice conceptual illustration. Adaptive learning rates ...
• 79.6k
Accepted

$\beta_1$ and $\beta_2$ are are the forgetting parameters for the Adam optimizer. The lower the either one, the faster the running average is updated (and hence the faster previous gradients are ...
• 156

### Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant

I think it is because you can see $\epsilon s/(\sqrt{r} + \delta)$ as an effective learning rate whose components corresponding to large second order moments are decreased and/ or small first order ...
• 184

### How well should I expect Adam to work?

There might be several factors at play here: The optimal learning rate for momentum based algorithms is usually lower than that for plain GD, because momentum increases the effective step size. I ...
• 23.2k

### When to use weight decay for ADAM optimiser?

Weight decay is a form of regularization that changes the objective function. You can also use other regularization techniques if you’d like. Either way, weight decay does alter the values used to ...
• 6,952