# Deep Learning: How does beta_1 and beta_2 in the Adam Optimizer affect it's learning?

My impression is that beta1 and beta2 affects how much the adam optimizer 'remember' it's previous movements, and my guess is that if the training isn't performing well, these values should be decreased, am I right?

However, what is the general intuition one should have when deciding whether to tweak these parameters or not, or by how much to change these values?

• Are you referring to the Adam Optimizer presented in this paper arxiv.org/pdf/1412.6980.pdf? It does have $\beta_1$ and $\beta_2$ parameters. If so, it might help to edit this into your question! Mar 6, 2017 at 4:48
• sorry I forgot to put this information in! I have edited the information and thank you for your suggestion! Mar 6, 2017 at 5:30

The hyper-parameters $$\beta_1$$ and $$\beta_2$$ of Adam are initial decay rates used when estimating the first and second moments of the gradient, which are multiplied by themselves (exponentially) at the end of each training step (batch). Based on my read of Algorithm 1 in the paper, decreasing $$\beta_1$$ and $$\beta_2$$ of Adam will make the learning slower, so if training is going too fast, that could help.

People using Adam might set $$\beta_1$$ and $$\beta_2$$ to high values (above 0.9) because they are multiplied by themselves (i.e., exponentially) during training. Setting $$\beta_1$$ and/or $$\beta_2$$ of Adam below 0.5 will result in drastic decreases as the number of training steps/batches increases.

My experience is that $$\beta_1$$ and $$\beta_2$$ of Adam are definitely tweakable and can be tuned via Bayesian optimization. For that, I use Optuna and have found both of them need to be set much lower than the defaults for optimal MNIST classification (based on maximizing test data classification accuracy) with a deep network trained on 3,833 samples (randomly chosen) from the training data set and validated on the remainder (I explain my reason for choosing that size of training set in this question).

My experiment using Adam to classify MNIST (without allowing amsgrad) revealed the following settings maximized categorical accuracy on the test data (10,000 images, never used for either training or validation) after 500 trials. For the study, I implemented median pruning after the first 100 trials, and allowed each trial a maximum 500 epochs to either converge (defined as patience of 50 epochs with no validation categorical cross-entropy decrease greater than 0.0001), or after pruning starts, beat the median validation of all trials up to that point and subsequently converge as defined above.

• single hidden layer followed by single max pooling layer
• square kernel size 2
• batch size 1024
• initial Adam learn rate ~ 0.026
• $$\beta_1$$ ~ 0.28
• $$\beta_2$$ ~ 0.93

I have posted updated demo code necessary to replicate my experiment; please replicate and post whether your results were similar. NB I am currently testing whether Adabelief will outperform Adam...that code is also posted, along with code for both Adam and Adabelief that attempts to implement Jeffreys priors for continuous hyper-parameters on only the first trial of the study (and fails to beat Optuna's native trial.suggest in the case of Adam).

• In comments, OP mentions that they've read the paper. Can you edit your answer to answer OP's questions and clarify how to choose values of $\beta_1$, $\beta_2$ so that the training of a neural network improves?
– Sycorax
Nov 30, 2020 at 16:48
• @Sycorax let me know if I missed worse, or hit closer to, the mark with edits. Dec 1, 2020 at 14:51

Simply put Betas are used as for smoothing the path to the convergence also providing some momentum to cross a local minima or saddle point. Now how does it affects can be thought of as Beta default value 0.9 just averages the gradient or (square of the gradient) of previous 10 batches which is calculated by 1/(1-0.9) which means it represent how we are coming from last 10 batches down when minimizing the loss. So instead of tuning that you can always tune the batch size it will have almost same impact!