# Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $$g$$ in momentum method to extract local linear trend from the statistics.

Then e.g. Adagrad, ADAM family adds averages of $$g_i\cdot g_i$$ to strengthen underrepresented coordinates.

TONGA can be seen as another step: updates $$g_i \cdot g_j$$ averages $$(gg^T$$) to model (non-centered) covariance matrix of gradients for Newton-like step.

What other exponential moving averages are worth to considered for SGD convergence, e.g. can be found in literature?

For example updating 4 exponential moving averages: of $$g$$, $$x$$, $$g\,x$$, $$x^2$$ gives MSE fitted parabola in a given direction, estimated Hessian $$H=\textrm{Cov}(g,x)\cdot \textrm{Cov}(x,x)^{-1}$$ in multiple directions (derivation). Analogously we could MSE fit e.g. in a single direction degree 3 polynomial if updating 6 averages: of $$g$$, $$x$$, $$gx$$, $$x^2$$, $$g\,x^2$$, $$x^3$$.

Have you seen such additional updated averages in literature, especially of $$g\,x$$? Is it worth e.g. to expand momentum method by such additional averages to model parabola in its direction for smarter step size?