I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also read about second order methods, and despite of the better convergence rate, it is sometimes mentioned that there are problems, for which second order methods are much better than SGD as SGD get stuck at some point or converges very slowly.
Unfortunately I couldn't find much information on that.
Does anyone know examples for which SGD has problems? Or do you know articles that tell something about this topic? Also articles, that only explain, why SGD gets stuck sometimes would be great.