# Questions tagged [adam]

An adaptive algorithm for gradient-based optimization of stochastic objective functions, often used to train deep neural networks.

68 questions
Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

### Stochastic objective function in Adam [duplicate]

Relating to the question What is a stochastic objective function?. In the paper of the Adam Algorithm they also mention a Stochastic objective function when explaining the algorithm. In this context I ...
0 votes
0 answers
12 views

### When is RMSprop Gradient Descent better than Momentum Gradient Descent

Line of thought The weight updates in RMSprop gradient descent can be considered as normalizing a gradient $d\Theta_i$ by the factor $1/(\sqrt{Sd\Theta_i} + \epsilon)$. If $\nabla_\Theta J$ of the ...
• 83
5 votes
1 answer
236 views

### What is a random variable in ADAM optimizer?

Look at the definition of ADAM optimizer, from the original paper of Kingma and Ba (paper): See algorithm $1$ for pseudo-code of our proposed algorithm Adam. Let $f(\theta)$ be a noisy objective ...
• 163
0 votes
0 answers
11 views

### Why Exponential Weighted Average in RMSProp

In optimization algorithms such as RMSProp, Adam, etc. why is an exponential weighted average used over a straight sum and divide? Seemingly an exponential weighted average would be useful when the ...
0 votes
0 answers
20 views

### Shrinkage / L1 regularization as a loss term versus a constraint (post-process step) with momentum optimizers

I have a complex model with very non-linear operations (divisions, exponentials, matrix inversions, square roots, Cholesky decompositions, etc...) for which I want to optimize the parameters. However, ...
0 votes
0 answers
3 views

1 vote
0 answers
96 views

### spikes in training CNN using Adam

I have an instability issue when training CNN on high dimensional data. I train Unet-3D model (https://arxiv.org/pdf/1505.04597.pdf) with batch normalization layer before each convolution layer. My ...
• 11
4 votes
2 answers
2k views

### Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
• 151
1 vote
0 answers
187 views

### Pathological curvature

How common is the issue of pathological curvature? I have been reading a post on the internet about it, and I would like to know how common this happens with deep learning models. Could you also ...
8 votes
2 answers
1k views

### Training a neural network on chess data

I have been writing a chess engine with a friend and the engine itself is really good already (2700+ CCRL). We had the idea to use a neural network to have a better evaluation of positions. Input to ...
• 281
2 votes
0 answers
940 views

### Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
• 3,528
1 vote
1 answer
521 views

### What is a stochastic objective function?

In the paper "ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION" they write in the introduction "Often, objective functions are stochastic." I know what a stochastic gradient descent is, but what does it ...
• 155
1 vote
0 answers
44 views

### Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
• 11
0 votes
1 answer
36 views

### Do we have guarantees about Adam's convergence when we reach an region with gradient $0$?

Recall the Adam update rule: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ \hat{v}_t = \dfrac{v_t}{1 - \...
• 147
1 vote
1 answer
142 views

### step size in the first epochs of adam are too large [closed]

When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below. This looks like ...
• 131
0 votes
0 answers
747 views

### understanding bias correction in ADAM algorithm [duplicate]

why is possible to approximate E[gi^2] with E[gt^2]? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different ...
0 votes
1 answer
79 views

### Different optimization behaviours on delta vs non-delta targets

I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ...
• 111
1 vote
0 answers
72 views

### SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
• 166
1 vote
1 answer
1k views

### Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
• 4,942
8 votes
2 answers
2k views

### How well should I expect Adam to work?

I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat. Well, when I implement the ...
8 votes
1 answer
5k views

### What does decay_steps mean in Tensorflow tf.train.exponential_decay?

I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ...
• 255
2 votes
2 answers
3k views

### Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant

In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate". To my understanding an adaptive learning rate should ...
• 528
1 vote
0 answers
717 views

### Adam optimizer's escape from local minima [duplicate]

I noticed some interesting behaviour in my loss history while training my model. Please note the sudden change in test loss at around epoch 106. A similar drop will appear around epoch ~1000. It ...
• 111
0 votes
1 answer
2k views

### How to select parameters for ADAM gradient descent

I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball ...
• 185
12 votes
0 answers
773 views

### what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
3 votes
1 answer
1k views

### Understanding a derivation of bias correction for the Adam optimizer

I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations ...
1 vote
0 answers
88 views

### Adaptive Moment Estimate - What is meant by parameters of Adam optimizer are biased towards zero initially? [duplicate]

Let us consider the Adam optimizer with the equation given below: $w_{t + 1} = w_{t} - \frac{\eta \times \bar{m_t}} {\sqrt{\bar{v_t} + \epsilon}}$ Here $w$ denotes weight (in time $t$ and $t + 1$) ...
• 111
8 votes
1 answer
2k views

### What does Diagonal Rescaling of the gradients mean in ADAM paper?

I was reading the original paper on ADAM (Adam: A Method for Stochastic Optimization), which mentions: [...] invariant to diagonal rescaling of the gradients, [...] What does it mean? Also, ...
3 votes
1 answer
717 views

### Two large decreses in loss function with ADAM optimizer

I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below. The task itself if just trying to ...
• 133
1 vote
0 answers
69 views

### Decreasing accuracy of neural network after long training

I trained a MLP network in Tensorflow which reached at some point 98.66 % accuracy. I used the MNIST data set and five hidden layers of the following size: 4096, 2048, 4096, 2048, 4096. In Tensorflow ...
• 615
7 votes
2 answers
2k views

### Etymology of "Adam" algorithm for gradient descent

What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization?
• 2,271
2 votes
0 answers
338 views

### Do we need to normalize inputs when using adaptive learning rate optimizers?

As stated in this thread Why do we need to normalize the images before we put them into CNN?, normalization of the inputs to a deep network to have zero mean and unit-deviation will make SGD more ...
• 932
4 votes
0 answers
448 views

### Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
• 217
6 votes
1 answer
2k views

### What should we do when changing SGD optimizer to Adam optimizer?

Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by ...
• 929
5 votes
1 answer
2k views

### Does RMSProp/Adam solve vanishing gradient problem?

RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ...
• 151
0 votes
1 answer
2k views

### XGBoost Optimization Function

Which optimization technique does XGBoost use? I have trained my XGBoost classifier using the 'binary:logistic' objective. Does it use SGD / Adam?
• 181