Problems, which are difficult for SGD

Question

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also read about second order methods, and despite of the better convergence rate, it is sometimes mentioned that there are problems, for which second order methods are much better than SGD as SGD get stuck at some point or converges very slowly.

Unfortunately I couldn't find much information on that.

Does anyone know examples for which SGD has problems? Or do you know articles that tell something about this topic? Also articles, that only explain, why SGD gets stuck sometimes would be great.

Answer 1

Second order methods use more information about the loss function (it computes two orders of derivative instead of only one), and so it approximates it better and has a better convergence.

It doesn't help that much with getting over local minima, but it should take fewer steps to converge.

The reason for which it is not used for neural nets is that every step has a complexity of $\mathcal O(p^2)$, where $p$ is the number of parameters, that's why we use it for linear models (few parameters), but not for deep learning (many parameters), it gets impractical to evaluate the whole Hessian matrix, and simple SGD is faster.

Answer 2

there are three common problems: 1.that if the loss changes quickly in one direction and slowly in other direction(in this case if we choose a small step size there will be very small progress on our training and if we choose large one this will cause over shooting and there will be need for get back in the steps)

2.local minima :which means that there is a point on a function where the function has zero gradient but it's not actually the bottom of the function

3.saddle point :it's close to the local minima in the meaning but it have a condition that 1 direction has to be different than others.