6
$\begingroup$

"Some commenters mentioned that the authors already use Adam and batchnorm, and Adam and batchnorm already are approximating second order behavior". [this really nice blog]

I can see how (correct me if I am wrong) Nesterov accelerated optimization (in some sense) solves an online convex optimization, where we choose which previously observed gradients to pick using Follow The Regularized Leader (FTRL, FoReL). But I can not see how adaptiveness emulates using second order statistics. And the statement about batchnorm is even more mysterious to me.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

It's kind of an imprecise statement, so It's hard to give a firm answer. Momentum and normalisation methods such as Adam, (diagonal-)AdaGrad and batch-normalization are (effectively) using diagonal approximations to the Hessian. Obviously, that's a very crude approximation, but it is approximating second-order (hessian) information.

I would associate second order methods with estimation of curvature, which is not something that can be done with diagonal approximations. IMHO it's too strong a statement to say they are approximating second order information.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ right; my point was that articles about BFGS method, for example, explicitly discuss this "diagonal approximation of Hessian", but I was not able to find an easy explanation of how does momentum and normalization archive that $\endgroup$
    – MInner
    Commented Apr 6, 2017 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.