This article states about the use of momentum in gradient descent for neural networks:

A lower bound, courtesy of Nesterov [5], states that momentum is, in a certain very narrow and technical sense, optimal.

I cannot open the Nesterov reference, since it is a book (and probably I'd find it hard to understand anyway).

Could you explain in what sense momentum is optimal for optimization?

  • $\begingroup$ The article you referenced is a great article from a really cool site. $\endgroup$ – jeffalltogether Dec 7 '18 at 4:54

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.