I am currently reading paper Auto-encoding Variational Bayes and I am not being able to understand the highlighted part in the screenshot below:

enter image description here

I am not understanding why there is f(z) and what is its significance. Also, why the gradient estimator exhibits high variance. It may be because I do not have knowledge on Monte Carlo gradient estimation. Can anyone please provide me some insight on this or provide me some helpful resources for understanding Monte Carlo Gradient Estimation?


1 Answer 1


Before answering your question, I would suggest you to read An Introduction to Variational Autoencoders which is a more detailed and extended version of the referenced paper.

So $f$ is used as a general term for an objective function which we aim to optimize.

Monte Carlo Methods use repeated sampling from random processes to estimate a value. This means that we draw $L$ latent variables $z^{(l)}$ from $q_{\theta}(z|x^{(i)})$ and then take the average of the gradient ($\frac{1}{L}\sum_{l=1}^L$).

In this case with $z$ being iid, the total variance is the sum of variances. Therefore, the variance increases when $L$ increases (and this only for one datapoint). In the next section, they say that $L$ can be set to 1 if the batch size is large enough.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.