# How can stochastic gradient descent avoid the problem of a local minimum?

I know that stochastic gradient descent has random behavior, but I don't know why.

-

The stochastic gradient (SG) algorithm behaves like a simulated annealing (SA) algorithm, where the learning rate of the SG is related to the temperature of SA. The randomness or noise introduced by SG allows to escape from local minima to reach a better minimum. Of course, it depends on how fast you decrease the learning rate. Read section 4.2, of Stochastic Gradient Learning in Neural Networks (pdf), where it is explained in more detail.

-
Don't oveelook Section 4.1as well, where the second theorem is for a limited case of nonconvex functions, saying it only converges (with infinite samples) to some point with gradient 0. It may not be global minimum or can even be a maximum. SGD is more interesting for more practical reasons such as distributed learning, not surely that it will "avoid" the local minimum. –  Lin Mar 22 at 8:25