There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$.

But we don't need full Hessian - there is already valuable information in second order model for a low dimensional subspace. For example just maintaining parabola model in a single direction: e.g. from momentum or ADAM method, might improve its choice of step size, and such online parabola model can be cheaply maintained e.g. by just updating 4 averages $(\theta, g,\theta g, \theta^2)$. Is there a successful parabola-enhanced line search?

enter image description here

Using second order model in $d\leq D$ dimensional subspace, the additional step cost generally grows like $\approx d^2$, allowing to simultaneously optimize in all these directions. However, it is said that neural network landscape is usually flat in most of directions - we need to chose somehow the locally more interesting directions.

Some basic questions here:

  • For what dimensional subspace it seems the most beneficial to use 2nd order model? For example: full Hessian, or layer-wise like in K-FAC, or e.g. a few dimensions like in saddle-free Newton, or in just a single direction enhancing first order method, or maybe none? If in a few, are there some hints how to choose this dimension?

  • How to optimally choose such locally promising $d$-dimensional subspace - where second order model seems the most beneficial? Krylov subspace like in saddle-free Newton, or maybe better some online-PCA of recent gradients, or something different?

  • Is local dependence important - is it sufficient to sometimes stop and estimate Hessian, or maybe it is better to go toward online methods: frequently updating the model?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.