0
$\begingroup$

I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball such that the parameters are sparse. Currently, learning rate = 0.2, $\beta_1$ = 0.99 and $\beta_2$ = 0.999, I am getting below graph for convergenceenter image description here

When I change $\beta_1$ to 0.999, I get the below graph enter image description here

I am not quite understanding how changes in learning rate, $\beta_1$ and $\beta_2$ affects the convergence, how can I make my convergence smoother. Also, when I am trying to make the parameters sparse, I get very slow convergence rate. How does making final solution sparse impact the convergence rate? Should I use other gradient descent methods?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.