Second order methods use more information about the loss function (it computes two orders of derivative instead of only one), and so it approximates it better and has a better convergence.

It doesn't help that much with getting over local minima, but it should take less steps to converge.

The reason for which it is not used for neural nets is that every step has a complexity of $\mathcal O(p^2)$, where $p$ is the number of parameters, that's why we use it for linear models (few paramters), but not for deep learning (many parameters), it gets impractical to evaluate the whole hessian matrix, and simple SGD is faster.

Second order methods use more information about the loss function (it computes two orders of derivative instead of only one), and so it approximates it better and has a better convergence.

It doesn't help that much with getting over local minima, but it should take fewer steps to converge.

The reason for which it is not used for neural nets is that every step has a complexity of $\mathcal O(p^2)$, where $p$ is the number of parameters, that's why we use it for linear models (few parameters), but not for deep learning (many parameters), it gets impractical to evaluate the whole Hessian matrix, and simple SGD is faster.