Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
My impression is that beta1 and beta2 affects how much the adam optimizer 'remember' it's previous movements, and my guess is that if the training isn't performing well, these values should be decreased, am I right?
However, what is the general intuition one should have when deciding whether to tweak these parameters or not, or by how much to change these values?
The hyper-parameters $\beta_1$ and $\beta_2$ of Adam are initial decay rates used when estimating the first and second moments of the gradient, which are multiplied by themselves (exponentially) at the end of each training step (batch). Based on my read of Algorithm 1 in the paper, decreasing $\beta_1$ and $\beta_2$ of Adam will make the learning slower, so if training is going too fast, that could help.
People using Adam might set $\beta_1$ and $\beta_2$ to high values (above 0.9) because they are multiplied by themselves (i.e., exponentially) during training. Setting $\beta_1$ and/or $\beta_2$ of Adam below 0.5 will result in drastic decreases as the number of training steps/batches increases.
My experience is that $\beta_1$ and $\beta_2$ of Adam are definitely tweakable and can be tuned via Bayesian optimization. For that, I use Optuna and have found both of them need to be set much lower than the defaults for optimal MNIST classification (based on maximizing test data classification accuracy) with a deep network trained on 3,833 samples (randomly chosen) from the training data set and validated on the remainder (I explain my reason for choosing that size of training set in this question).
My experiment using Adam to classify MNIST (without allowing amsgrad) revealed the following settings maximized categorical accuracy on the test data (10,000 images, never used for either training or validation) after 500 trials. For the study, I implemented median pruning after the first 100 trials, and allowed each trial a maximum 500 epochs to either converge (defined as patience of 50 epochs with no validation categorical cross-entropy decrease greater than 0.0001), or after pruning starts, beat the median validation of all trials up to that point and subsequently converge as defined above.
I have posted updated demo code necessary to replicate my experiment; please replicate and post whether your results were similar. NB I am currently testing whether Adabelief will outperform Adam...that code is also posted, along with code for both Adam and Adabelief that attempts to implement Jeffreys priors for continuous hyper-parameters on only the first trial of the study (and fails to beat Optuna's native .trial.suggest in the case of Adam)
Simply put Betas are used as for smoothing the path to the convergence also providing some momentum to cross a local minima or saddle point. Now how does it affects can be thought of as Beta default value 0.9 just averages the gradient or (square of the gradient) of previous 10 batches which is calculated by 1/(1-0.9) which means it represent how we are coming from last 10 batches down when minimizing the loss. So instead of tuning that you can always tune the batch size it will have almost same impact!
