Evidence lower bound (ELBO) can be maximized, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the minimization of the corresponding Kullback–Leibler divergence. The paper that introduces variational auto-encoders (VAEs) uses the SGVB (stochastic gradient variational Bayes) formula rather than ELBO. What is the difference between these?


1 Answer 1


The evidence lower bound is a bound on the log probability of the data. But there is no straightforward way to compute the ELBO, since it requires taking an expectation over the variational posterior. Therefore we need a procedure to estimate the ELBO (more specifically we need some way to estimate the gradient of the ELBO so we can optimize it).

The straightforward method is simply to estimate the expectation by sampling from the variational posterior, and then to compute the gradient of the estimator using the score function gradient estimator. However the variance of this method is too high for practical use, which is why the authors introduce their "SGVB" estimator.

  • $\begingroup$ Can you explain why the SGVB estimator has a lower variance than the score function gradient? $\endgroup$
    – 900edges
    Mar 15, 2021 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.