I have a doubt in adagrad.

The update rule in adagrad is like this:

theta = theta - delta*alpha/sqrt(G)

where, G = sum of squares of historical gradients.

delta = current gradient

and alpha is initial learning rate and sqrt G is supposed to decay it.

But if gradients are less always than 1, than this will have a boosting effect on alpha. Is this ok?

This will continue till sum of squares dont reach 1(which might take some time). Once it reaches 1, alpha gets damped.

Should I use something like below?

theta = theta - delta*alpha/(1+sqrt(G))

1 Answer 1


Yes, it is reasonable to have this boosting effect.

This effect means that (to steal the words of C. Dyer) the learning algorithm "'pays attention' to rare but informative features". It is the direct consequence of the key insight behind adagrad that the dimension of the gradient vector are unequally informative. If you make the addition you propose you will dampen the learning in the case of rare feature updates.

Aside the manuscript already linked, I found these notes by E. Fox quite helpful too. The comment on the geometric interpretation of the algorithm.

  • $\begingroup$ What should I divide with in the first example update then if I don't add 1? the denominator will be 0 in that case. $\endgroup$
    – Swapniel
    Commented Nov 25, 2015 at 4:47
  • $\begingroup$ Sorry for this naive question. From second update onwards we don't have $I$ in $G$? I mean we add gradient squares to previous $G$ right? $\endgroup$
    – Swapniel
    Commented Nov 25, 2015 at 5:33
  • $\begingroup$ Sorry my bad, $G$ should be $\sum_{\tau=1}^t g_\tau g_\tau^T$ where $g$ is the observed subgradient vector. For your first step just calculate numerically at one of the subgradients at $x_0$ and you should be fine. The whole issue is about concatenating the new sub-gradients to your $g$. $\endgroup$
    – usεr11852
    Commented Nov 25, 2015 at 5:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.