3
$\begingroup$

I am searching momentum. And below animated gif shows one advantage of momentum. And the momentum is expressed like this:

$\mathbf{W}^{k+1}$ = $\mathbf{W}^{k}$ – η $\frac {\partial E}{\partial \mathbf{W}}$ + γ$\mathbf{W}^{k – 1}$

enter image description here

My question is why can the momentum escape from the saddle point after losing the magnitude of the gradient, namely although the momentum is about to be almost settled while the SGD can't escape. I read this discussion and I googled a lot. So, in some case the momentum can escape from a saddle point while the normal SGD can't. However, in this animated gif I can't explain why the momentum can escape.

Could anyone explain this?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ Why is SGD (Stochastic Gradient Descent) listed with these other things? SGD is about how to calculate the Gradient, not about how to update it. I don't think SGD's should get stuck in a saddle, because any subset that you choose to calculate the stochastic gradient will have a slightly different error plane, so it should never stop updating (unless you decrease the learning rate to 0) $\endgroup$
    – Sam
    Commented Oct 19, 2017 at 14:42
  • $\begingroup$ I totally agree with you, so do you think the animated gif is strange, too? $\endgroup$
    – Kazuya Tomita
    Commented Oct 20, 2017 at 0:17
  • $\begingroup$ The momentum doesn't escape. It's the green line and it stays in the saddle point until the end. $\endgroup$
    – Aaron
    Commented Oct 20, 2017 at 2:13
  • $\begingroup$ Yeah, I think the Gif is strange $\endgroup$
    – Sam
    Commented Oct 20, 2017 at 13:32
  • 1
    $\begingroup$ Fixing @Sam's comment, SGD is about parameters updating!, it's an optimizer! The backpropagation algorithm is what computes the gradients. $\endgroup$
    – Rafael Toledo
    Commented Jun 1, 2022 at 20:57

1 Answer 1

Reset to default
2
$\begingroup$

Finally, I concluded that the animated GIF is strange for some reasons(see our comments linking my question).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.