Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics.

Then e.g. Adagrad, ADAM family adds averages of $g_i\cdot g_i$ to strengthen underrepresented coordinates.

TONGA can be seen as another step: updates $g_i \cdot g_j$ averages $(gg^T$) to model (non-centered) covariance matrix of gradients for Newton-like step.

What other exponential moving averages are worth to considered for SGD convergence, e.g. can be found in literature?

For example updating 4 exponential moving averages: of $g$, $x$, $g\,x$, $x^2$ gives MSE fitted parabola in a given direction, estimated Hessian $H=\textrm{Cov}(g,x)\cdot \textrm{Cov}(x,x)^{-1}$ in multiple directions (derivation). Analogously we could MSE fit e.g. in a single direction degree 3 polynomial if updating 6 averages: of $g$, $x$, $gx$, $x^2$, $g\,x^2$, $x^3$.

Have you seen such additional updated averages in literature, especially of $g\,x$? Is it worth e.g. to expand momentum method by such additional averages to model parabola in its direction for smarter step size?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.