An adaptive algorithm for gradient-based optimization of stochastic objective functions, often used to train deep neural networks.

44 questions
Filter by
Sorted by
Tagged with
18 views

Recall the Adam update rule: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ $$\hat{v}_t = \dfrac{v_t}{1 - \... 0answers 12 views ### Stochastic/batch gradient descent (type SGD/ADAM) with weighted mean square error loss Assume I assign uneven weights to losses of different examples, i.e. I set my SGD/ADAM to train a universal approximator f (e.g. a neural net) by minimization of a weighted mean square error:  L = \... 1answer 18 views ### step size in the first epochs of adam are too large [closed] When I train models in keras with keras.optimizers.Adam(learning_rate=0.001), I typically get a history of the training error over the training time in epochs like in the plot below. This looks like ... 0answers 50 views ### understanding bias correction in ADAM algorithm [duplicate] why is possible to approximate E[gi^2] with E[gt^2]? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different ... 1answer 61 views ### Different optimization behaviours on delta vs non-delta targets I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ... 0answers 13 views ### Learning rate Adam Algorithm [duplicate] Finding the 'optimal' learning rate for regular gradient descent is generally a pretty important task, as it could influence the results a lot. I was wondering whether this is the case for the ADAM ... 0answers 14 views ### SGD versus Adamax on XOR operator I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ... 1answer 555 views ### Combining Random forest with Adam (or an other gradient method) There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)... 2answers 571 views ### How well should I expect Adam to work? I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat. Well, when I implement the ... 0answers 121 views ### Significant oscillation in fully connected layer neural network training I am using neural networks for a regression problem. There is one output variable. 80 features per sample. 2000 samples total. I have 2 hidden fully connected layers with 500, and 100 neurons, ... 1answer 2k views ### What does decay_steps mean in Tensorflow tf.train.exponential_decay? I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ... 2answers 616 views ### Why we call ADAM an a adaptive learning rate algorithm if the step size is a constant In the book "Deep Learning" by Goodfellow et.al, the ADAM algorithm is described in sub-chapter 8.5 "Algorithm with Adaptive Learning Rate". To my understanding an adaptive learning rate should ... 0answers 264 views ### Adam optimizer's escape from local minima [duplicate] I noticed some interesting behaviour in my loss history while training my model. Please note the sudden change in test loss at around epoch 106. A similar drop will appear around epoch ~1000. It ... 1answer 316 views ### How to select parameters for ADAM gradient descent I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, \beta_1 and \beta_2. Along with gradient descent, I am projecting the paramters on L_1 ball ... 0answers 210 views ### what is the mistake of convergence proof in Adam Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ... 1answer 363 views ### Understanding a derivation of bias correction for the Adam optimizer I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations ... 0answers 51 views ### Adaptive Moment Estimate - What is meant by parameters of Adam optimizer are biased towards zero initially? [duplicate] Let us consider the Adam optimizer with the equation given below: w_{t + 1} = w_{t} - \frac{\eta \times \bar{m_t}} {\sqrt{\bar{v_t} + \epsilon}} Here w denotes weight (in time t and t + 1) ... 1answer 642 views ### What does Diagonal Rescaling of the gradients mean in ADAM paper? I was reading the original paper on ADAM (Adam: A Method for Stochastic Optimization), which mentions: [...] invariant to diagonal rescaling of the gradients, [...] What does it mean? Also, ... 1answer 318 views ### Two large decreses in loss function with ADAM optimizer I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below. The task itself if just trying to ... 0answers 34 views ### Decreasing accuracy of neural network after long training I trained a MLP network in Tensorflow which reached at some point 98.66 % accuracy. I used the MNIST data set and five hidden layers of the following size: 4096, 2048, 4096, 2048, 4096. In Tensorflow ... 2answers 1k views ### Etymology of “Adam” algorithm for gradient descent What is the history behind the choice of the name "Adam" as used in Adam: A Method for Stochastic Optimization? 0answers 166 views ### Do we need to normalize inputs when using adaptive learning rate optimizers? As stated in this thread Why do we need to normalize the images before we put them into CNN?, normalization of the inputs to a deep network to have zero mean and unit-deviation will make SGD more ... 0answers 258 views ### Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD) I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ... 1answer 523 views ### What should we do when changing SGD optimizer to Adam optimizer? Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by ... 1answer 425 views ### Does RMSProp/Adam solve vanishing gradient problem? RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ... 1answer 1k views ### XGBoost Optimization Function Which optimization technique does XGBoost use? I have trained my XGBoost classifier using the 'binary:logistic' objective. Does it use SGD / Adam? 1answer 22k views ### RMSProp and Adam vs SGD I am performing experiments on the EMNIST validation set using networks with RMSProp, Adam and SGD. I am achieving 87% accuracy with SGD(learning rate of 0.1) and dropout (0.1 dropout prob) as well ... 1answer 3k views ### The reason of superiority of Limited-memory BFGS over ADAM solver I am using Multilayer Perceptron MLPClassifier for training a classification model for my problem. I noticed that using the solver lbfgs (I guess it implies Limited-... 1answer 3k views ### No change in accuracy using Adam Optimizer when SGD works fine I have been training a Spatial Transformer network with DNN on GTRSB dataset. I initially used SGF with momentum and was able to achieve good accuracy. For further improvements and testing, I ... 2answers 8k views ### How does batch size affect Adam Optimizer? What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer? From what I understood (I might be ... 0answers 512 views ### Loss update with mini batches and after epoch If I understand correctly, when using deep learning with mini batches, we have a forward and backward pass in every mini batch (with the corresponding optimizer). But does something different happen ... 2answers 8k views ### Explanation of Spikes in training loss vs. iterations with Adam Optimizer I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam ... 0answers 1k views ### Spikes in loss function of deep neural network training [duplicate] I'm training a simple feed-forward Deep Network with standard relu/sigmoid activation functions, using Tensorflow. I tried several optimizers and I noticed that those belonging to the Adam family show ... 2answers 2k views ### Why update all parameters at each step of the Adam optimiser even when we have sparse observations? Adam is a method for stochastic optimisation. The algorithm is given below. Consider our parameters that we wish to optimise$$\boldsymbol{\theta} = [\theta_1, \theta_2]$$observations$$\...
1k views

I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that ...
2k views

### Is manually tuning learning rate during training redundant with optimization methods like Adam?

I have seen some high-profile deep learning papers where an optimization method like Adam was used, yet the learning rate was manually changed at specific iterations. What is the relationship between ...
682 views

### Convergence proof of ADAM optimizer

I've read ADAM paper (https://arxiv.org/abs/1412.6980) and found out a suspicious part. In Lemma 10.3 of Appendix, \begin{align*} \sqrt{\|g_{1:T,i}\|_2^2 - g_{T,i}^2} & \le \|g_{1:T,i}\|_2 - \...
8k views

### How to choose between SGD with Nesterov momentum and Adam?

I'm currently implementing a neural network architecture on Keras. I would like to optimize the training time, and I'm considering using alternative optimizers such as SGD with Nesterov Momentum and ...
2k views

### Training accuracy increase abruptly at first epoch to 99%. is it normal?

I expected my training process is like this: But my training accuracy increased to 99% at only 1 epoch, not steadily increase. I was suspicious about high learning rate, so i used various learning ...
474 views

Optimization algorithms like Adagrad and ADAM decay your learning rate over time. To me this sounds like a bad idea for online training since you're always getting new data as opposed to retraining on ...
7k views

### Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Begnio, Goodfellow and Courtville: Adam includes bias corrections to ...
8k views

### What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville: Adam is generally regarded as ...