# What happens when gradient in adagrad is less than 1 at each step?

I have a doubt in adagrad.

The update rule in adagrad is like this:

theta = theta - delta*alpha/sqrt(G)


where, G = sum of squares of historical gradients.

delta = current gradient

and alpha is initial learning rate and sqrt G is supposed to decay it.

But if gradients are less always than 1, than this will have a boosting effect on alpha. Is this ok?

This will continue till sum of squares dont reach 1(which might take some time). Once it reaches 1, alpha gets damped.

Should I use something like below?

theta = theta - delta*alpha/(1+sqrt(G))


## 1 Answer

Yes, it is reasonable to have this boosting effect.

This effect means that (to steal the words of C. Dyer) the learning algorithm "'pays attention' to rare but informative features". It is the direct consequence of the key insight behind adagrad that the dimension of the gradient vector are unequally informative. If you make the addition you propose you will dampen the learning in the case of rare feature updates.

Aside the manuscript already linked, I found these notes by E. Fox quite helpful too. The comment on the geometric interpretation of the algorithm.

• What should I divide with in the first example update then if I don't add 1? the denominator will be 0 in that case. Commented Nov 25, 2015 at 4:47
• Sorry for this naive question. From second update onwards we don't have $I$ in $G$? I mean we add gradient squares to previous $G$ right? Commented Nov 25, 2015 at 5:33
• Sorry my bad, $G$ should be $\sum_{\tau=1}^t g_\tau g_\tau^T$ where $g$ is the observed subgradient vector. For your first step just calculate numerically at one of the subgradients at $x_0$ and you should be fine. The whole issue is about concatenating the new sub-gradients to your $g$. Commented Nov 25, 2015 at 5:49