Skip to main content

Questions tagged [adam]

An adaptive algorithm for gradient-based optimization of stochastic objective functions, often used to train deep neural networks.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Adam's $\beta_1$ fixed in practice but required to depend on $t$ for convergence proofs

In the paper ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, the exponential moving average parameter $\beta_1$ is set to $0.9$ as default in most ML/DL APIs but the convergence proof requires that $\...
Toonia's user avatar
  • 13
0 votes
1 answer
828 views

What is the Llama2 number of steps? [closed]

Llama2 is pretrained with 2 trillion of tokens: $2\times10^9$, and its batch size is of $4\times 10^6$. We can calculate the number of steps (times we upgrade the parameters) per epoch as follows: $$\...
Noether's user avatar
1 vote
0 answers
18 views

How to prove that my neural network satisfies convergence assumptions of the Adam optimizer?

Recently, I read a paper (this paper) that provides convergence analysis of the Adam optimizer without assuming the smoothness condition of the loss function. Suppose I have a neural network $f(\theta)...
poglhar's user avatar
  • 51
1 vote
0 answers
71 views

How to prove the smoothness assumption of the loss function in the convergence of the Adam optimizer?

I have read 2 papers (this and this) about the convergence of the Adam optimizer. One of the assumptions is the smoothness of the loss function, meaning that the gradient of the loss function is ...
poglhar's user avatar
  • 51
1 vote
2 answers
55 views

t in adam optimizer [closed]

I'm newbie in NN and I get confused about frequency t in adam optimizer. if I have NN with architecture like this: Can't it be said that Xk (k=1,2,..., n), where n is the number of input data? in the ...
Andryan's user avatar
  • 47
2 votes
1 answer
148 views

Turning point of Adam optimizer for LSTM

I'm training a two-layer LSTM with Adam optimizer for time-series data. I encountered several times that there was a "turning point" of the MAE vs. epochs plot. Is this a normal behavior?
lovetl2002's user avatar
1 vote
0 answers
108 views

Which is better: performance stability or high performance in the initial epochs but extremely unstable performance during validation?

I am working on a multiclassification problem using time series data. I am using a hybrid model (such as LSTM, CNN, attention, etc.). I tried two optimizers, ADAM with 0.001 learning rate and SGD with ...
Ahmad's user avatar
  • 161
1 vote
0 answers
68 views

When is RMSprop Gradient Descent better than Momentum Gradient Descent

Can you give an example where RMSprop is better than momentum? In what cases does RMSprop (i.e. scaling $d\Theta_i$) help, but Momentum does not? How is the combination of Momentum and RMSprop (i.e. ...
DaveFar's user avatar
  • 93
5 votes
1 answer
501 views

What is a random variable in ADAM optimizer?

Look at the definition of ADAM optimizer, from the original paper of Kingma and Ba (paper): See algorithm $1$ for pseudo-code of our proposed algorithm Adam. Let $f(\theta)$ be a noisy objective ...
Lucian's user avatar
  • 203
0 votes
0 answers
122 views

Why Exponential Weighted Average in RMSProp

In optimization algorithms such as RMSProp, Adam, etc. why is an exponential weighted average used over a straight sum and divide? Seemingly an exponential weighted average would be useful when the ...
lafarrjam99's user avatar
2 votes
0 answers
251 views

ADAM bias correction derivation

Where $\beta_2\in [0,1)$ and $g_i$ is the gradient at step i. We approximate E[gi^2] with E[gt^2] by a correction term $\zeta$. I think this is because it is assumed that the gradients are bounded. Is ...
Le Ploit's user avatar
2 votes
1 answer
8k views

Why does Adam optimizer seems to prevail over Nadam optimizer? [closed]

I have been studying the way Adam optimizer works, and how it combines both RMSProp and Momentum optimizers. So the following question arises: Why not combine Nesterov Accelerated Gradient together ...
mespinosa's user avatar
2 votes
1 answer
338 views

Adam optimizer with distributed training

Does Adam optimizer applicable to either method of distributed training BSP, ASP, SSP or it can be used with synchronous mode only ?
Elena Gurevich's user avatar
3 votes
1 answer
1k views

Why does ADAM optimization perform well on non-convex functions and bad on convex functions?

I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
Amaan's user avatar
  • 237
1 vote
0 answers
62 views

How should I set $\vec{\mu}$ in NAdam optimization?

In Dozat 2016 they introduce a sequence of hyperparameters $\mu_0, \cdots, \mu_T$ where $T$ is the total number of iterations. Naturally $T$ is dependent on the convergence of the parameters, so it ...
Galen's user avatar
  • 9,361
2 votes
1 answer
193 views

Increasing training set decreases accuracy

I've ~16000 labeled data. I split it in ~8000 for training and ~8000 for testing of my RBF neural network, find the best hyperparameters (RMSE from 1.2 to 1.4*) and finally train the model on the ...
dpelisek's user avatar
  • 143
0 votes
0 answers
397 views

How should i expect exponential decay to work in this case? (Adam optimizer)

I was facing high learning rate issues i.e., validation loss started to diverge after 9-13 epochs. In order mitigate that i have significantly reduced the learning rate from 4e-3 to 4e-4 and ...
Luciano Dourado's user avatar
0 votes
1 answer
44 views

When do Adaptive Optimization Algorithms modify their parameters?

When do "Ada" optimizers (e.g. Adagrad, Adam, etc...) "adapt" their parameters? Is it at the end of each mini-batch or epoch?
Marsellus Wallace's user avatar
3 votes
0 answers
202 views

Why adversarial training prefer SGD over Adam?

Why adversarial training methods (e.g. Trades) use SGD as optimizer rather than Adam?
cmhzc's user avatar
  • 31
3 votes
1 answer
2k views

When to use weight decay for ADAM optimiser?

If you use weight decay for gradient descent (ADAM specifically) do you need to use regularisation for loss function? I believe the answer is yes since the gradient descent involves the ...
Robert Lewis's user avatar
2 votes
1 answer
367 views

Gradient based optimization of step function w.r.t number of steps

I am trying to optimize the parameter b in the following simple function using gradient descent in PyTorch: $$ y = \frac{\lfloor{xb} \rfloor + 0.5}{b} $$ x is in $[0,1]$ and b is continuous and in $[5,...
Winfried Lötzsch's user avatar
1 vote
0 answers
511 views

spikes in training CNN using Adam

I have an instability issue when training CNN on high dimensional data. I train Unet-3D model (https://arxiv.org/pdf/1505.04597.pdf) with batch normalization layer before each convolution layer. My ...
Zuser's user avatar
  • 11
6 votes
2 answers
7k views

Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
Opal's user avatar
  • 171
1 vote
0 answers
467 views

Pathological curvature

How common is the issue of pathological curvature? I have been reading a post on the internet about it, and I would like to know how common this happens with deep learning models. Could you also ...
Davi Sena's user avatar
12 votes
2 answers
2k views

Training a neural network on chess data

I have been writing a chess engine with a friend and the engine itself is really good already (2700+ CCRL). We had the idea to use a neural network to have a better evaluation of positions. Input to ...
Finn Eggers's user avatar
6 votes
1 answer
2k views

Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
Josh's user avatar
  • 4,518
2 votes
1 answer
1k views

What is a stochastic objective function?

In the paper "ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION" they write in the introduction "Often, objective functions are stochastic." I know what a stochastic gradient descent is, but what does it ...
Amit Keinan's user avatar
1 vote
0 answers
167 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
Simon's user avatar
  • 111
0 votes
1 answer
85 views

Do we have guarantees about Adam's convergence when we reach an region with gradient $0$?

Recall the Adam update rule: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ $$\hat{v}_t = \dfrac{v_t}{1 - \...
actinidia's user avatar
  • 145
1 vote
1 answer
289 views

step size in the first epochs of adam are too large [closed]

When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below. This looks like ...
Jakob's user avatar
  • 131
1 vote
0 answers
1k views

understanding bias correction in ADAM algorithm [duplicate]

why is possible to approximate E[gi^2] with E[gt^2]? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different ...
Kethan Chauhan's user avatar
0 votes
1 answer
101 views

Different optimization behaviours on delta vs non-delta targets

I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ...
Gal Avineri's user avatar
1 vote
0 answers
126 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
Carlos Porta's user avatar
1 vote
1 answer
2k views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
Ferdi's user avatar
  • 5,197
9 votes
2 answers
3k views

How well should I expect Adam to work?

I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat. Well, when I implement the ...
Joseph Barnett's user avatar
8 votes
1 answer
6k views

What does decay_steps mean in Tensorflow tf.train.exponential_decay?

I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ...
Suleka_28's user avatar
  • 255
5 votes
2 answers
5k views

Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant

In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate". To my understanding an adaptive learning rate should ...
S12000's user avatar
  • 578
1 vote
0 answers
722 views

Adam optimizer's escape from local minima [duplicate]

I noticed some interesting behaviour in my loss history while training my model. Please note the sudden change in test loss at around epoch 106. A similar drop will appear around epoch ~1000. It ...
VegardKT's user avatar
  • 111
0 votes
1 answer
5k views

How to select parameters for ADAM gradient descent

I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball ...
newbie's user avatar
  • 225
19 votes
1 answer
1k views

what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
Виталик Бушаев's user avatar
4 votes
1 answer
2k views

Understanding a derivation of bias correction for the Adam optimizer

I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations ...
Виталик Бушаев's user avatar
1 vote
0 answers
94 views

Adaptive Moment Estimate - What is meant by parameters of Adam optimizer are biased towards zero initially? [duplicate]

Let us consider the Adam optimizer with the equation given below: $w_{t + 1} = w_{t} - \frac{\eta \times \bar{m_t}} {\sqrt{\bar{v_t} + \epsilon}}$ Here $w$ denotes weight (in time $t$ and $t + 1$) ...
skr's user avatar
  • 111
13 votes
1 answer
3k views

What does Diagonal Rescaling of the gradients mean in ADAM paper?

I was reading the original paper on ADAM (Adam: A Method for Stochastic Optimization), which mentions: [...] invariant to diagonal rescaling of the gradients, [...] What does it mean? Also, ...
thelogicalkoan's user avatar
3 votes
1 answer
927 views

Two large decreses in loss function with ADAM optimizer

I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below. The task itself if just trying to ...
McAngus's user avatar
  • 133
1 vote
0 answers
109 views

Decreasing accuracy of neural network after long training

I trained a MLP network in Tensorflow which reached at some point 98.66 % accuracy. I used the MNIST data set and five hidden layers of the following size: 4096, 2048, 4096, 2048, 4096. In Tensorflow ...
Gilfoyle's user avatar
  • 659
7 votes
2 answers
2k views

Etymology of "Adam" algorithm for gradient descent

What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization?
Tom Hale's user avatar
  • 2,611
4 votes
0 answers
595 views

Do we need to normalize inputs when using adaptive learning rate optimizers?

As stated in this thread Why do we need to normalize the images before we put them into CNN?, normalization of the inputs to a deep network to have zero mean and unit-deviation will make SGD more ...
hirschme's user avatar
  • 1,130
7 votes
1 answer
956 views

Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
flxh's user avatar
  • 257
6 votes
1 answer
3k views

What should we do when changing SGD optimizer to Adam optimizer?

Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by ...
mining's user avatar
  • 1,019
6 votes
1 answer
3k views

Does RMSProp/Adam solve vanishing gradient problem?

RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ...
Siyuan Ren's user avatar