Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value

$Г_{t + 1} = \frac{\sqrt{V_{t+1}}}{a_{t+1}} - \frac{\sqrt{V_t}}{a_t}$, where $V$ is moving average of squared gradients and $a$ is learning rate,

is presumed to be positive, however, it's only true for SGD and AdaGrad, while for RMSProp and Adam can be anything. I can't find a place where this property would be presumed and used in adam's origin convergence proof, can someone point it out to me ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.