I'm studying variational inference and trying to implement the algorithm. But I came across a question when computing the ELBO.

Suppose for the latent variable $z$, I have a variational distribution $q_w(z)$, $w$ is the parameter I want to infer. The joint distribution of data and $z$ is given as $P(Data, z)$. So the ELBO is

$ELBO = \int_z q_w(z)\log P(Data, z) d_z - \int_z q_w(z)\log q_w(z)d_z$.

When I'm implementing the algorithm for the first part $\int_z q_w(z)\log P(Data, z) d_z$, I know that I'm going to draw a sample $r$ from $q_w(z)$ first. But do I need to go back to compute the likelihood $q_w(r)$, then use the likelihood times $\log P(Data, r)$? Or I can simply compute $\log P(Data, r)$ and ignore the $q_w(r)$ term?


In the Monte Carlo estimation principle we deal with intractable integrals by means of sampling. In particular, the integral $$ L := \int q_w(z) \log P(\text{Data}, z) dz $$is too hard to compute. Luckily, it has the form of an average of some function (namely, $\log P(\text{Data}, \cdot)$) with weights $q_w(z)$. We thus can form a stochastic estimate of this quantity: $$ \hat{L}_M := \frac{1}{M} \sum_{m=1}^M \log P(\text{Data}, z_m), \quad \text{where} \;\; z_1, \dots, z_M \stackrel{i.i.d.}{\sim} q_w(z) $$

Now, $\hat{L}_M$ is a random variable, whereas $L$ originally was a number. However, we can show that $\hat{L}_M$ revolves around $L$, namely, expected value of $\hat{L}_M$ is equal to $L$ (for any $M$), and the variance (a measure of how much we tend to miss $L$) decreases as we take more samples $M$: $$ \mathbb{E}\left[\hat{L}_M\right] = L, \quad\quad \mathbb{V}\left[\hat{L}_M\right] = \frac{1}{M} \mathbb{V}\left[\log P(\text{Data}, z_1)\right] $$ Obviously, the bigger the $M$ is – the more precise estimate of $L$ you get. But this come at a price of your algorithm become increasingly computationally expensive (you'd need to perform $M$ independent calculations!). In stochastic optimization people often use $M = 1$ and deal with the incurred variance with (1) decreasing learning rate, (2) performing more optimization steps.

Notice the absence of any weighting factors $q_w(z)$ in $\hat{L}_M$. This is because these weights have been incorporated by the sampling procedure. The bigger the weight $q_w(r)$ for some $r$ – the more often this particular $r$ (or, more precisely, values from its vicinity) will be realized when sampled.

Finally, Monte Carlo method is, I'd say, a fundamental tool in many scientific and engineering disciplines. I encourage you to learn more (you can start with the wikipedia).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.