# Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also read about second order methods, and despite of the better convergence rate, it is sometimes mentioned that there are problems, for which second order methods are much better than SGD as SGD get stuck at some point or converges very slowly.

Unfortunately I couldn't find much information on that.

Does anyone know examples for which SGD has problems? Or do you know articles that tell something about this topic? Also articles, that only explain, why SGD gets stuck sometimes would be great.

The reason for which it is not used for neural nets is that every step has a complexity of $$\mathcal O(p^2)$$, where $$p$$ is the number of parameters, that's why we use it for linear models (few parameters), but not for deep learning (many parameters), it gets impractical to evaluate the whole Hessian matrix, and simple SGD is faster.