How does Stochastic Gradient Descent with momentum distinguish between local minima and global minima? I have several questions regarding this.


*

*How does SGD momentum know to converge at global minina and skip over local minima?

*I read that "SGD momentum goes past the minima (due to its velocity build up) and then correct themselves and then comes back towards minima". Is this correction due to the friction term? 
Why does it correct only for global minima but not for local minima/saddle points? 

 A: First of all, it's important to correct the statements here. A momentum based SGD does not necessarily skip local minima and converge to global minima. Rather, it is very likely to skip small bumps and more likely to spend time around big bumps. The basic idea of this argument is that global minima are more likely to be big bumps. But more precisely, the solutions at the critical points around big bumps is more likely to be nearly equivalent (in terms of loss) to the global minima than the solution at critical points around small bumps. To be clear, note that this argument requires some unproven statements about local and global minima (i.e., relative sizes of bumps around critical point). 
To illustrate, consider the plot below. 

In this case, we have a global minima at x = 4, and two local minima at x = 0, 10. Note that the domain of attraction for the local minima is much smaller for the two non-global critical points, while it is much larger for the global minima. So the idea is that a momentum and/or stochastic gradient descent is less likely to get stuck in a small domain of attraction than a more vanilla gradient method since (a) stochastic methods "bounce" around the critical point much more than vanilla gradient methods, so they can more easily "bounce" out of a small domain than a large domain and (b) momentum based methods are more likely to just move through a small domain without stopping, since instead of switching directions when they are on the other side of a critical point but still in the domain of attraction, they only decrease their momentum. If they don't decrease their momentum enough to switch directions before leaving the domain of attraction, they will pass over the critical point. 
Note that this is making many assumptions about the domain of attraction to critical points. Instead, consider if the function looked like this:

In this case, the domain of attraction to the global min (4) is smaller than the domain of attraction to the local min. So our argument above does not apply to this function!
