I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also read about second order methods, and despite of the better convergence rate, it is sometimes mentioned that there are problems, for which second order methods are much better than SGD as SGD get stuck at some point or converges very slowly.

Unfortunately I couldn't find much information on that.

Does anyone know examples for which SGD has problems? Or do you know articles that tell something about this topic? Also articles, that only explain, why SGD gets stuck sometimes would be great.


Second order methods use more information about the loss function (it computes two orders of derivative instead of only one), and so it approximates it better and has a better convergence.

It doesn't help that much with getting over local minima, but it should take fewer steps to converge.

The reason for which it is not used for neural nets is that every step has a complexity of $\mathcal O(p^2)$, where $p$ is the number of parameters, that's why we use it for linear models (few parameters), but not for deep learning (many parameters), it gets impractical to evaluate the whole Hessian matrix, and simple SGD is faster.

  • $\begingroup$ Thanks for your answer. Maybe my question was not precise enough. I know that the complexity is often quadratic, but there are also methods, which use an approximate Hessian and are more efficient. My question is, if there are problems, for which it is worth the effort, as SGD gets stuck at some point or converges so slowly that it takes to long to reach the minimum. At my university course it was mentioned that such problems exist, but I can't find literature on this topic. So I thought maybe someone of you knows something about this topic. $\endgroup$ – Lisa Sep 23 '20 at 8:34
  • $\begingroup$ Try basic SGD on these and see how it performs: en.wikipedia.org/wiki/Test_functions_for_optimization $\endgroup$ – Quantoisseur Sep 28 '20 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.