"Some commenters mentioned that the authors already use Adam and batchnorm, and Adam and batchnorm already are approximating second order behavior". [this really nice blog]
I can see how (correct me if I am wrong) Nesterov accelerated optimization (in some sense) solves an online convex optimization, where we choose which previously observed gradients to pick using Follow The Regularized Leader (FTRL, FoReL). But I can not see how adaptiveness emulates using second order statistics. And the statement about batchnorm is even more mysterious to me.