# Why isn't "Saddle-Free Newton" descent algorithm used in practice?

Recently I have read a paper by Yann Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, where they introduce an interesting descent algorithm called Saddle-Free Newton, which seems to be exactly tailored for neural network optimization and shouldn't suffer from getting stuck at saddle points like first order methods as vanilla SGD.

The paper dates back into 2014, so it's nothing brand new, however, I haven't seen it being used "in the wild". Why is this method not being used? Is the Hessian computation too prohibitive for real world sized problems/networks? Is there even some open source implementation of this algorithm, possibly to be used with some of the major deep learning frameworks?

Update Feb 2019: there is an implementation available now: https://github.com/dave-fernandes/SaddleFreeOptimizer)

• Good question, I couldn't find anything. However, the pseudocode is very simple so you could give it a try yourself, in which case there are some useful implementation details in one of the authors' doctoral thesis (page 103, papyrus.bib.umontreal.ca/xmlui/bitstream/handle/1866/13710/…) Jun 3, 2017 at 17:59
• I found reference to this same paper in an Uber Deep-Neuroevolution Blog post. Link: eng.uber.com/deep-neuroevolution You might ask the author if they have any implementation online / shared via GitHub. Dec 18, 2017 at 23:35
• here is an implementation for TensorFlow: github.com/dave-fernandes/SaddleFreeOptimizer Feb 4, 2019 at 17:00
• If I had to guess, my assumption would be that computing + inverting the Hessian is impractical when your model has millions of parameters.
– Sycorax
Feb 4, 2019 at 17:09
• Can you refine your question from "is there an implementation"? That seems to afford, yes/no answers &/or sounds like a software request (which is off topic here). Could your question be elaborated into something like, 'what difficulties explain why there don't seem to have been more implementations'? Feb 4, 2019 at 17:25