# Why does momentum escape from a saddle point in this famous image?

I am searching momentum. And below animated gif shows one advantage of momentum. And the momentum is expressed like this:

$\mathbf{W}^{k+1}$ = $\mathbf{W}^{k}$ – η $\frac {\partial E}{\partial \mathbf{W}}$ + γ$\mathbf{W}^{k – 1}$ My question is why can the momentum escape from the saddle point after losing the magnitude of the gradient, namely although the momentum is about to be almost settled while the SGD can't escape. I read this discussion and I googled a lot. So, in some case the momentum can escape from a saddle point while the normal SGD can't. However, in this animated gif I can't explain why the momentum can escape.

Could anyone explain this?

• Why is SGD (Stochastic Gradient Descent) listed with these other things? SGD is about how to calculate the Gradient, not about how to update it. I don't think SGD's should get stuck in a saddle, because any subset that you choose to calculate the stochastic gradient will have a slightly different error plane, so it should never stop updating (unless you decrease the learning rate to 0) – Sam Oct 19 '17 at 14:42
• I totally agree with you, so do you think the animated gif is strange, too? – Kazuya Tomita Oct 20 '17 at 0:17
• The momentum doesn't escape. It's the green line and it stays in the saddle point until the end. – Aaron Oct 20 '17 at 2:13
• Yeah, I think the Gif is strange – Sam Oct 20 '17 at 13:32