1
$\begingroup$

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example.

$\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}}\times g_{t,i}$.

Here $G_{t,ii}$ represents the summation of previous gradients. Why is it so that the square root is so important? If the reason is related to the fact that $G_{t,ii}$ will become very large and hence will prohibit learning, is it not possible to get the same effect using another hyperparameter $\beta$ and using $\beta\times G_{t,ii}$?

Or will a linear scaling down not have the same effect as a non-linear scaling down?

Improve this question
$\endgroup$
3
  • $\begingroup$ Given how complex these systems are, the most likely answer that you'll get is "try it and see, but your results may not generalize to other cases." There are remarkably few results proving general things about neural networks or their optimization $\endgroup$ – generic_user Dec 14 '17 at 13:31
  • $\begingroup$ But the statement made in most articles about AdaGrad is that it almost always never works well when the square root is not taken, which seems like a very general statement. Quoting the Stanford Article, "Amusingly, the square root operation turns out to be very important and without it, the algorithm performs much worse." $\endgroup$ – Ameet Deshpande Dec 14 '17 at 15:03
  • $\begingroup$ Heck, why not add the exponent as yet another hyperparameter to tune. $\endgroup$ – generic_user Dec 14 '17 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.