The hyper-parameters $\beta_1$ and $\beta_2$ of Adam are initial decay rates used when estimating the first and second moments of the gradient, which are multiplied by themselves (exponentially) at the end of each training step (batch). Based on my read of Algorithm 1 in the paper, decreasing $\beta_1$ and $\beta_2$ of Adam will make the learning slower, so if training is going too fast, that could help.
People using Adam might set $\beta_1$ and $\beta_2$ to high values (above 0.9) because they are multiplied by themselves (i.e., exponentially) during training. Setting $\beta_1$ and/or $\beta_2$ of Adam below 0.5 will result in drastic decreases as the number of training steps/batches increases.
My experience is that $\beta_1$ and $\beta_2$ of Adam are definitely tweakable and can be tuned via Bayesian optimization. For that, I use Optuna and have found both of them need to be set
much lower than the defaults for optimal MNIST classification (based on maximizing test data classification accuracy) with a deep network trained on 3,833 samples (randomly chosen) from the training data set and validated on the remainder (I explain my reason for choosing that size of training set in this question).
My experiment using Adam to classify MNIST (without allowing
amsgrad) revealed the following settings maximized categorical accuracy on the test data (10,000 images, never used for either training or validation) after 500 trials. For the study, I implemented median pruning after the first 100 trials, and allowed each trial a maximum 500 epochs to either converge (defined as patience of 50 epochs with no validation categorical cross-entropy decrease greater than 0.0001), or after pruning starts, beat the median validation of all trials up to that point and subsequently converge as defined above.
- single hidden layer followed by single max pooling layer
- square kernel size 2
- batch size 1024
- initial Adam learn rate ~ 0.026
- $\beta_1$ ~ 0.28
- $\beta_2$ ~ 0.93
I have posted updated demo code necessary to replicate my experiment; please replicate and post whether your results were similar. NB I am currently testing whether Adabelief will outperform Adam...that code is also posted, along with code for both Adam and Adabelief that attempts to implement Jeffreys priors for continuous hyper-parameters on only the first trial of the study
(and fails to beat Optuna's native .
trial.suggest in the case of Adam)