For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension.
I am trying to understand second order SGD convergence methods (slides), and it seems like they often attract to a saddle, like natural gradient wanting to take us to a close point with zero gradient.
There are many approaches trying to escape non-convexity by approximating Hessian with some positive definite matrix, for example:
Gauss-Newton method and Fisher information matrix using some linear approximation,
TONGA uses covariance matrix of recent gradients.
While such approximation tries to pretend that minimized function is locally convex, in fact it isn't - we can be near a saddle in this moment.
How do such positive Hessian approximations handle saddles?
For example, naively, covariance matrix of recent gradients should be similar near minimum and near saddle (ignores sign of curvature) - why using it doesn't attract to saddles?
