Questions tagged [adam]
An adaptive algorithm for gradient-based optimization of stochastic objective functions, often used to train deep neural networks.
58
questions
2
votes
1answer
48 views
Increasing training set decreases accuracy
I've ~16000 labeled data. I split it in ~8000 for training and ~8000 for testing of my RBF neural network, find the best hyperparameters (RMSE from 1.2 to 1.4*) and finally train the model on the ...
0
votes
0answers
24 views
How can the scores of CV done with training data better then the score of all training data
I have the following issue, when I use the adam solver (MLPRegressor from sklaern) my cross validation (10 repeats a 5 splits) metrics (r2, maxError, RMSE, MAE) are all better (r2 ~ 0.94) as the ...
0
votes
0answers
16 views
How should i expect exponential decay to work in this case? (Adam optimizer)
I was facing high learning rate issues i.e., validation loss started to diverge after 9-13 epochs. In order mitigate that i have significantly reduced the learning rate from 4e-3 to 4e-4 and ...
0
votes
0answers
14 views
Can a variable batch-size be problematic?
I am currently facing an issue with one of the models I am working on (it's a Transducer model but that does not really matter for the question in general). The problem is that the model sometimes ...
0
votes
1answer
24 views
When do Adaptive Optimization Algorithms modify their parameters?
When do "Ada" optimizers (e.g. Adagrad, Adam, etc...) "adapt" their parameters? Is it at the end of each mini-batch or epoch?
2
votes
0answers
39 views
Why adversarial training prefer SGD over Adam?
Why adversarial training methods (e.g. Trades) use SGD as optimizer rather than Adam?
1
vote
1answer
181 views
When to use weight decay for ADAM optimiser?
If you use weight decay for gradient descent (ADAM specifically) do you need to use regularisation for loss function?
I believe the answer is yes since the gradient descent involves the ...
2
votes
1answer
59 views
Gradient based optimization of step function w.r.t number of steps
I am trying to optimize the parameter b in the following simple function using gradient descent in PyTorch:
$$ y = \frac{\lfloor{xb} \rfloor + 0.5}{b} $$
x is in $[0,1]$ and b is continuous and in $[5,...
0
votes
0answers
37 views
proper reference for the lazy adam optimizer
ADAM is the most cited paper in the machine learning literature. There are many variants to it. However a highly practical variant, LazyAdam, aimed at improving performance under sparse weights ...
1
vote
0answers
33 views
spikes in training CNN using Adam
I have an instability issue when training CNN on high dimensional data.
I train Unet-3D model (https://arxiv.org/pdf/1505.04597.pdf) with batch normalization layer before each convolution layer.
My ...
0
votes
0answers
30 views
Adam optimizer in sPacy giving strange results
I have the following toy example to play with the TextCategorizer CNN model in sPacy. Given the distribution of my input dataset, I am expecting the test results to ...
3
votes
2answers
606 views
Adam (adaptive) optimizer(s) learning rate tuning
I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate:
The learning is arguably the ...
1
vote
0answers
100 views
Pathological curvature
How common is the issue of pathological curvature? I have been reading a post on the internet about it, and I would like to know how common this happens with deep learning models.
Could you also ...
9
votes
2answers
1k views
Training a neural network on chess data
I have been writing a chess engine with a friend and the engine itself is really good already (2700+ CCRL). We had the idea to use a neural network to have a better evaluation of positions.
Input to ...
1
vote
0answers
496 views
Momentum vs Polyak averaging
I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers.
For example, is it correct to say that ...
0
votes
1answer
291 views
What is a stochastic objective function?
In the paper "ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION" they write in the introduction "Often, objective functions are stochastic."
I know what a stochastic gradient descent is, but what does it ...
1
vote
0answers
25 views
Adam converges while SGD does not improve at all
I am trying to build a model based movie recommendation system with a neural network.
The architecture looks as follows:
...
0
votes
1answer
27 views
Do we have guarantees about Adam's convergence when we reach an region with gradient $0$?
Recall the Adam update rule:
$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$
$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$
$$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$
$$\hat{v}_t = \dfrac{v_t}{1 - \...
1
vote
1answer
74 views
step size in the first epochs of adam are too large [closed]
When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below.
This looks like ...
0
votes
0answers
496 views
understanding bias correction in ADAM algorithm [duplicate]
why is possible to approximate E[gi^2] with E[gt^2]? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different ...
0
votes
1answer
72 views
Different optimization behaviours on delta vs non-delta targets
I built a simple classification model that is required to predict a probability distribution for a set of two available classes.
The target distributions are not necessarily delta distributions. i.e ...
0
votes
0answers
38 views
SGD versus Adamax on XOR operator
I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code:
...
1
vote
1answer
1k views
Combining Random forest with Adam (or an other gradient method)
There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD?
Can I use Adam (Adaptive moment estimation)...
6
votes
2answers
1k views
How well should I expect Adam to work?
I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat.
Well, when I implement the ...
6
votes
1answer
5k views
What does decay_steps mean in Tensorflow tf.train.exponential_decay?
I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ...
2
votes
2answers
2k views
Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant
In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate".
To my understanding an adaptive learning rate should ...
1
vote
0answers
651 views
Adam optimizer's escape from local minima [duplicate]
I noticed some interesting behaviour in my loss history while training my model.
Please note the sudden change in test loss at around epoch 106.
A similar drop will appear around epoch ~1000.
It ...
0
votes
1answer
1k views
How to select parameters for ADAM gradient descent
I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball ...
8
votes
0answers
612 views
what is the mistake of convergence proof in Adam
Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
3
votes
1answer
958 views
Understanding a derivation of bias correction for the Adam optimizer
I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations ...
1
vote
0answers
78 views
Adaptive Moment Estimate - What is meant by parameters of Adam optimizer are biased towards zero initially? [duplicate]
Let us consider the Adam optimizer with the equation given below:
$w_{t + 1} = w_{t} - \frac{\eta \times \bar{m_t}} {\sqrt{\bar{v_t} + \epsilon}}$
Here $w$ denotes weight (in time $t$ and $t + 1$) ...
8
votes
1answer
1k views
What does Diagonal Rescaling of the gradients mean in ADAM paper?
I was reading the original paper on ADAM (Adam: A Method for Stochastic Optimization), which mentions:
[...] invariant to diagonal rescaling of the gradients, [...]
What does it mean?
Also, ...
3
votes
1answer
645 views
Two large decreses in loss function with ADAM optimizer
I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below.
The task itself if just trying to ...
1
vote
0answers
46 views
Decreasing accuracy of neural network after long training
I trained a MLP network in Tensorflow which reached at some point 98.66 % accuracy. I used the MNIST data set and five hidden layers of the following size: 4096, 2048, 4096, 2048, 4096. In Tensorflow ...
7
votes
2answers
1k views
Etymology of "Adam" algorithm for gradient descent
What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization?
2
votes
0answers
290 views
Do we need to normalize inputs when using adaptive learning rate optimizers?
As stated in this thread Why do we need to normalize the images before we put them into CNN?, normalization of the inputs to a deep network to have zero mean and unit-deviation will make SGD more ...
4
votes
0answers
387 views
Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)
I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam.
SPSA is intended for noisy objective functions, and Adam for ...
4
votes
1answer
2k views
What should we do when changing SGD optimizer to Adam optimizer?
Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by ...
5
votes
1answer
1k views
Does RMSProp/Adam solve vanishing gradient problem?
RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ...
0
votes
1answer
2k views
XGBoost Optimization Function
Which optimization technique does XGBoost use? I have trained my XGBoost classifier using the 'binary:logistic' objective. Does it use SGD / Adam?
19
votes
1answer
41k views
RMSProp and Adam vs SGD
I am performing experiments on the EMNIST validation set using networks with RMSProp, Adam and SGD. I am achieving 87% accuracy with SGD(learning rate of 0.1) and dropout (0.1 dropout prob) as well ...
16
votes
2answers
7k views
The reason of superiority of Limited-memory BFGS over ADAM solver
I am using Multilayer Perceptron MLPClassifier for training a classification model for my problem. I noticed that using the solver lbfgs (I guess it implies Limited-...
3
votes
1answer
7k views
No change in accuracy using Adam Optimizer when SGD works fine
I have been training a Spatial Transformer network with DNN on GTRSB dataset. I initially used SGF with momentum and was able to achieve good accuracy.
For further improvements and testing, I ...
8
votes
2answers
22k views
How does batch size affect Adam Optimizer?
What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?
From what I understood (I might be ...
1
vote
0answers
603 views
Loss update with mini batches and after epoch
If I understand correctly, when using deep learning with mini batches, we have a forward and backward pass in every mini batch (with the corresponding optimizer). But does something different happen ...
24
votes
2answers
27k views
Explanation of Spikes in training loss vs. iterations with Adam Optimizer
I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam ...
5
votes
0answers
2k views
Spikes in loss function of deep neural network training [duplicate]
I'm training a simple feed-forward Deep Network with standard relu/sigmoid activation functions, using Tensorflow. I tried several optimizers and I noticed that those belonging to the Adam family show ...
2
votes
2answers
3k views
Why update all parameters at each step of the Adam optimiser even when we have sparse observations?
Adam is a method for stochastic optimisation. The algorithm is given below.
Consider our parameters that we wish to optimise
$$\boldsymbol{\theta} = [\theta_1, \theta_2]$$
observations
$$\...
0
votes
0answers
2k views
Better Performance With Gradient Descent than Adam on word2vec
I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that ...
9
votes
1answer
3k views
Is manually tuning learning rate during training redundant with optimization methods like Adam?
I have seen some high-profile deep learning papers where an optimization method like Adam was used, yet the learning rate was manually changed at specific iterations.
What is the relationship between ...
