theta = theta - delta*alpha/sqrt(G)


where, G = sum of squares of historical gradients.

and alpha is initial learning rate and sqrt G is supposed to decay it.

But if gradients are less always than 1, than this will have a boosting effect on alpha. Is this ok?

This will continue till sum of squares dont reach 1(which might take some time). Once it reaches 1, alpha gets damped.

Should I use something like below?

theta = theta - delta*alpha/(1+sqrt(G))


• Sorry for this naive question. From second update onwards we don't have $I$ in $G$? I mean we add gradient squares to previous $G$ right? – Swapniel Nov 25 '15 at 5:33
• Sorry my bad, $G$ should be $\sum_{\tau=1}^t g_\tau g_\tau^T$ where $g$ is the observed subgradient vector. For your first step just calculate numerically at one of the subgradients at $x_0$ and you should be fine. The whole issue is about concatenating the new sub-gradients to your $g$. – usεr11852 says Reinstate Monic Nov 25 '15 at 5:49