I am searching momentum. And below animated gif shows one advantage of momentum. And the momentum is expressed like this:

$\mathbf{W}^{k+1}$ = $\mathbf{W}^{k}$ – η $\frac {\partial E}{\partial \mathbf{W}}$ + γ$\mathbf{W}^{k – 1}$

enter image description here

My question is why can the momentum escape from the saddle point after losing the magnitude of the gradient, namely although the momentum is about to be almost settled while the SGD can't escape. I read this discussion and I googled a lot. So, in some case the momentum can escape from a saddle point while the normal SGD can't. However, in this animated gif I can't explain why the momentum can escape.

Could anyone explain this?

  • 4
    $\begingroup$ Why is SGD (Stochastic Gradient Descent) listed with these other things? SGD is about how to calculate the Gradient, not about how to update it. I don't think SGD's should get stuck in a saddle, because any subset that you choose to calculate the stochastic gradient will have a slightly different error plane, so it should never stop updating (unless you decrease the learning rate to 0) $\endgroup$
    – Sam
    Oct 19 '17 at 14:42
  • $\begingroup$ I totally agree with you, so do you think the animated gif is strange, too? $\endgroup$ Oct 20 '17 at 0:17
  • $\begingroup$ The momentum doesn't escape. It's the green line and it stays in the saddle point until the end. $\endgroup$
    – Aaron
    Oct 20 '17 at 2:13
  • $\begingroup$ Yeah, I think the Gif is strange $\endgroup$
    – Sam
    Oct 20 '17 at 13:32

Finally, I concluded that the animated GIF is strange for some reasons(see our comments linking my question).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.