I have been reading this paper [1] and [2]. There is a statement that reads as follows:

The variance between compressed gradient and uncompressed gradient must be made as small as possible because this way it has been shown that the learning is accelerated.

I infer that in general the smaller the variance of gradient in SGD, the faster the convergence. How can we prove this statement?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.