Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics.
Then e.g. Adagrad, ADAM family adds averages of $g_i\cdot g_i$ to strengthen underrepresented coordinates.
TONGA can be seen as another step: updates $g_i \cdot g_j$ averages $(gg^T$) to model (non-centered) covariance matrix of gradients for Newton-like step.
What other exponential moving averages are worth to considered for SGD convergence, e.g. can be found in literature?
For example updating 4 exponential moving averages: of $g$, $x$, $g\,x$, $x^2$ gives MSE fitted parabola in a given direction, estimated Hessian $H=\textrm{Cov}(g,x)\cdot \textrm{Cov}(x,x)^{-1}$ in multiple directions (derivation). Analogously we could MSE fit e.g. in a single direction degree 3 polynomial if updating 6 averages: of $g$, $x$, $gx$, $x^2$, $g\,x^2$, $x^3$.
Have you seen such additional updated averages in literature, especially of $g\,x$? Is it worth e.g. to expand momentum method by such additional averages to model parabola in its direction for smarter step size?
