# Question about the implementation of variational inference

I'm studying variational inference and trying to implement the algorithm. But I came across a question when computing the ELBO.

Suppose for the latent variable $$z$$, I have a variational distribution $$q_w(z)$$, $$w$$ is the parameter I want to infer. The joint distribution of data and $$z$$ is given as $$P(Data, z)$$. So the ELBO is

$$ELBO = \int_z q_w(z)\log P(Data, z) d_z - \int_z q_w(z)\log q_w(z)d_z$$.

When I'm implementing the algorithm for the first part $$\int_z q_w(z)\log P(Data, z) d_z$$, I know that I'm going to draw a sample $$r$$ from $$q_w(z)$$ first. But do I need to go back to compute the likelihood $$q_w(r)$$, then use the likelihood times $$\log P(Data, r)$$? Or I can simply compute $$\log P(Data, r)$$ and ignore the $$q_w(r)$$ term?

In the Monte Carlo estimation principle we deal with intractable integrals by means of sampling. In particular, the integral $$L := \int q_w(z) \log P(\text{Data}, z) dz$$is too hard to compute. Luckily, it has the form of an average of some function (namely, $$\log P(\text{Data}, \cdot)$$) with weights $$q_w(z)$$. We thus can form a stochastic estimate of this quantity: $$\hat{L}_M := \frac{1}{M} \sum_{m=1}^M \log P(\text{Data}, z_m), \quad \text{where} \;\; z_1, \dots, z_M \stackrel{i.i.d.}{\sim} q_w(z)$$
Now, $$\hat{L}_M$$ is a random variable, whereas $$L$$ originally was a number. However, we can show that $$\hat{L}_M$$ revolves around $$L$$, namely, expected value of $$\hat{L}_M$$ is equal to $$L$$ (for any $$M$$), and the variance (a measure of how much we tend to miss $$L$$) decreases as we take more samples $$M$$: $$\mathbb{E}\left[\hat{L}_M\right] = L, \quad\quad \mathbb{V}\left[\hat{L}_M\right] = \frac{1}{M} \mathbb{V}\left[\log P(\text{Data}, z_1)\right]$$ Obviously, the bigger the $$M$$ is – the more precise estimate of $$L$$ you get. But this come at a price of your algorithm become increasingly computationally expensive (you'd need to perform $$M$$ independent calculations!). In stochastic optimization people often use $$M = 1$$ and deal with the incurred variance with (1) decreasing learning rate, (2) performing more optimization steps.
Notice the absence of any weighting factors $$q_w(z)$$ in $$\hat{L}_M$$. This is because these weights have been incorporated by the sampling procedure. The bigger the weight $$q_w(r)$$ for some $$r$$ – the more often this particular $$r$$ (or, more precisely, values from its vicinity) will be realized when sampled.