13
$\begingroup$

Recently I have read a paper by Yann Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, where they introduce an interesting descent algorithm called Saddle-Free Newton, which seems to be exactly tailored for neural network optimization and shouldn't suffer from getting stuck at saddle points like first order methods as vanilla SGD.

The paper dates back into 2014, so it's nothing brand new, however, I haven't seen it being used "in the wild". Why is this method not being used? Is the Hessian computation too prohibitive for real world sized problems/networks? Is there even some open source implementation of this algorithm, possibly to be used with some of the major deep learning frameworks?

Update Feb 2019: there is an implementation available now: https://github.com/dave-fernandes/SaddleFreeOptimizer)

|cite|improve this question
$\endgroup$
  • $\begingroup$ Good question, I couldn't find anything. However, the pseudocode is very simple so you could give it a try yourself, in which case there are some useful implementation details in one of the authors' doctoral thesis (page 103, papyrus.bib.umontreal.ca/xmlui/bitstream/handle/1866/13710/…) $\endgroup$ – galoosh33 Jun 3 '17 at 17:59
  • 1
    $\begingroup$ I found reference to this same paper in an Uber Deep-Neuroevolution Blog post. Link: eng.uber.com/deep-neuroevolution You might ask the author if they have any implementation online / shared via GitHub. $\endgroup$ – Cantren Dec 18 '17 at 23:35
  • $\begingroup$ here is an implementation for TensorFlow: github.com/dave-fernandes/SaddleFreeOptimizer $\endgroup$ – Dave F Feb 4 '19 at 17:00
  • $\begingroup$ If I had to guess, my assumption would be that computing + inverting the Hessian is impractical when your model has millions of parameters. $\endgroup$ – Sycorax says Reinstate Monica Feb 4 '19 at 17:09
  • 1
    $\begingroup$ Can you refine your question from "is there an implementation"? That seems to afford, yes/no answers &/or sounds like a software request (which is off topic here). Could your question be elaborated into something like, 'what difficulties explain why there don't seem to have been more implementations'? $\endgroup$ – gung - Reinstate Monica♦ Feb 4 '19 at 17:25
2
$\begingroup$

Better optimization does not necessarily mean a better model. In the end what we care about is how well the model generalizes, and not necessarily how good the performance on the training set is. Fancier optimization techniques usually perform better and converge faster on the training set, but do not always generalize as well as basic algorithms. For example this paper shows that SGD can generalize better than ADAM optimizer. This can also be the case with some second order optimization algorithms.

[Edit] Removed the first point as it does not apply here. Thanks to bayerj for pointing this out.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ While I agree with the second point, the first one is not valid here. The authors propose to do optimisation only in the Krylov subspace, which does not require quadratic complexity. $\endgroup$ – bayerj Feb 28 '19 at 21:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.