There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$.
But we don't need full Hessian - there is already valuable information in second order model for a low dimensional subspace. For example just maintaining parabola model in a single direction: e.g. from momentum or ADAM method, might improve its choice of step size, and such online parabola model can be cheaply maintained e.g. by just updating 4 averages $(\theta, g,\theta g, \theta^2)$. Is there a successful parabola-enhanced line search?
Using second order model in $d\leq D$ dimensional subspace, the additional step cost generally grows like $\approx d^2$, allowing to simultaneously optimize in all these directions. However, it is said that neural network landscape is usually flat in most of directions - we need to chose somehow the locally more interesting directions.
Some basic questions here:
For what dimensional subspace it seems the most beneficial to use 2nd order model? For example: full Hessian, or layer-wise like in K-FAC, or e.g. a few dimensions like in saddle-free Newton, or in just a single direction enhancing first order method, or maybe none? If in a few, are there some hints how to choose this dimension?
How to optimally choose such locally promising $d$-dimensional subspace - where second order model seems the most beneficial? Krylov subspace like in saddle-free Newton, or maybe better some online-PCA of recent gradients, or something different?
Is local dependence important - is it sufficient to sometimes stop and estimate Hessian, or maybe it is better to go toward online methods: frequently updating the model?
