Im new to optimization so sorry if this question is ridiculous. In gradient descent/ascent based optimization, one big problem seems to be getting stuck in a local minimum randomly. Ways in which I understand optimizers try to alleviate this issue include lr scheduling (correct me if this is false), random initialization and ensembling. These methods are statistical in nature, but would it be possible to look at higher order gradients once stuck in a local minimum and get information about which neighboring minimum to hop to next? Would the computation be too expensive?
It is possible to avoid getting stuck at poor saddle points for unconstrained minimization either by randomly perturbing iterates, or by using higher-order information. See the related work section of this paper for details. As you have indicated in your question, this paper shows that gradient descent with random initialization converges to local minima under additional assumptions. However, it is generally NP-hard to even hope to converge to local minima, see this paper.