I'm trying to understand why ADAM uses the gradient and the gradient squared respectively as estimators the first and second moments. I was assuming that the random variable we were estimating was the true gradient, but I cannot reason why E[g] = g and E[g^2] = g^2. Can someone help me understand this? Thanks!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.