Recently I have read a paper by Yann Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, where they introduce an interesting descent algorithm called Saddle-Free Newton, which seems to be exactly tailored for neural network optimization and shouldn't suffer from getting stuck at saddle points like first order methods as vanilla SGD.
The paper dates back into 2014, so it's nothing brand new, however, I haven't seen it being used "in the wild". Why is this method not being used? Is the Hessian computation too prohibitive for real world sized problems/networks? Is there even some open source implementation of this algorithm, possibly to be used with some of the major deep learning frameworks?
Update Feb 2019: there is an implementation available now: https://github.com/dave-fernandes/SaddleFreeOptimizer)