I have a doubt in adagrad.
The update rule in adagrad is like this:
theta = theta - delta*alpha/sqrt(G)
where, G = sum of squares of historical gradients.
delta = current gradient
and alpha is initial learning rate and sqrt G is supposed to decay it.
But if gradients are less always than 1, than this will have a boosting effect on alpha. Is this ok?
This will continue till sum of squares dont reach 1(which might take some time). Once it reaches 1, alpha gets damped.
Should I use something like below?
theta = theta - delta*alpha/(1+sqrt(G))