6
$\begingroup$

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers.

For example, is it correct to say that in momentum one averages parameter gradients while in Polyak we average parameter values?

From what I gather, Adam uses bias-corrected, running

  • averages of gradients (1st moment)
  • second-order moments of gradients

Has the use of Polyak averaging been studied in combination with Adam? In what cases is it expected to help?

Improve this question
$\endgroup$
1
  • $\begingroup$ Classic momentum works a little like EWMA, where farther in the past has (much) lower weight, and it is done each iteration. It looks like Polyak is about equal weights or at least not decayed weights over a window of successive updates. If momentum is 0.8, then parameter weight for 5 steps ago is $0.8^{5} = 0.32$. If all values were equal then 5 steps ago would be 1/6 = 0.16. If the past has lower weights, learning might be faster. $\endgroup$
    – EngrStudent
    Commented Jul 13, 2020 at 22:40

1 Answer 1

Reset to default
1
$\begingroup$

For your second question, I just found a related paper (2020 Granziol D, Wan X, Roberts S) Gadam: Combining Adaptivity with Iterate Averaging Gives Greater Generalisation that probably provides an answer:

We introduce Gadam, which combines Adam and iterate averaging (IA) to significantly improve generalisation performance without sacrificing adaptivity.

This paper provides an approach to combine Polyak averaging (the IA therein) and Adam, discusses its strengths in shrinking original Adam's generalization error, and lists possible reasons that this line of research has not been much investigated.

For your first question, I would like to point out that Polyak averaging is developed solely for SGD to stabilizes the estimates. A simple average of historical estimates in a regular GD is meaningless. Meanwhile, the momentum can also be used in GD to alleviate the zig-zag behavior when the parameters are travelling in a deep valley of the optimization surface. From my perspective, there is not much to compare between momentum and Polyak averaging beyond that they are both used to accelerate the convergence. They are more of two independent techniques. To confirm this, note that Adam also includes a momentum term and it can be used in conjunction with Polyak averaging according the aforementioned paper.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Jul 6, 2023 at 5:34
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – Shawn Hemelstrand
    Commented Jul 6, 2023 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.