I am reading this paper on variational inference and this website.

One thing I am confused about is how they get to decompose ELBO, where $ELBO(q) = E_q[log~p(z,x)] - E_q[log~q(z)]$, when focusing on one latent variable's variational distribution $q_j$ like this:

$$ ELBO(q_j) = E_j[E_{-j}[log~p(z_j, z_{-j}, x)]] - E_j[log~q_j(z_j)] + C $$

They say that they use iterated expectation but I had a hard time decomposing $ELBO(q_j)$ using that ($E[X] = E[E[X|Y]]$).

Can anyone elaborate on this? Thanks!

UPDATE: $q(z)=\prod_{i} q(z_i)$ is an assumption so I understand the decomposition of the 2nd term.


2 Answers 2


Your update has stated that you are using the mean-field variational family, or in other words that $q(z)=\prod_{i} q(z_i)$ which means that $$ \log q(z) = \sum_i \log q(z_i) \tag{*}. $$

So \begin{align*} \text{ELBO}(q) &= E_q[\log p(z,x)] - E_q[\log q(z)] \\ &= E_q[\log p(z_j, z_{-j},x)] - E_q[\log q(z_j, z_{-j})] \\ &= E_q[\log p(z_j, z_{-j},x)] - E_q[\log q(z_j) + \sum_{i\neq j}\log q(z_i)] \tag{*}\\ &= E_q[E\left(\log p(z_j, z_{-j},x) \mid z_{j} \right)] - E_q[\log q(z_j)] - E\left[\sum_{i\neq j}\log q(z_i)\right].\\ \end{align*} This is equivalent to equation (19) in your first linked document.

  • $\begingroup$ Thanks for the reply! The last equality confuses me a bit: you said that the outer expectation is taken w.r.t. $z$ or $z_{j}$. But according to the law iterated expectation, the outer expectation should be taken w.r.t. $Y$, which is $z_{-j}$ here. Shouldn't the first term be $E_q [E_q[log~p(z_j, z_{-j}, x) \vert z_{j}]]$? $\endgroup$
    – Zhiya
    Feb 12, 2018 at 17:24
  • $\begingroup$ I am still confused. Why the equation above has the same term on both the RHS and LHS? $\endgroup$
    – Zhiya
    Feb 12, 2018 at 17:40
  • $\begingroup$ Thanks for the clarification but I could not get your idea. I figured out one understanding below. Let me know if you think that's appropriate or not if you have time. Thanks again! $\endgroup$
    – Zhiya
    Feb 12, 2018 at 18:00
  • $\begingroup$ @Zhiya looks good! $\endgroup$
    – Taylor
    Feb 12, 2018 at 21:10

I read those two materials again and think that the decomposition may be done via the following rearrangement:

\begin{align} ELBO & = E_q[log~p(x,z)] - E_q[log~q(z)] \\ & = E_q[log~p(x, z_j, z_{-j})] - \sum_{q_l}E_{q_l}[q_l(z_l)] \\ & [\text{Here we use the fact that } q(z) \text{ can be factorized}]\\ & = E_j\Big[E_{-j}\big[ log~p(x, z_j, z_{-j}) \vert z_{j} \big] \Big] - E_{q_j}[q_j] + const \\ \end{align}

Now, according to the definition of expectation, we have: \begin{align} E_{-j}\big[ log~p(x, z_j, z_{-j}) \vert z_{j} \big] &= \int_{-j} log~p(x, z_j, z_{-j})~q(z_{-j}|z_j) dq_{-j} \\ & = \int_{-j} log~p(x, z_j, z_{-j})~q(z_{-j}) dq_{-j} \\ & = E_{-j}\big[ log~p(x, z_j, z_{-j}) \big] \end{align} [We assume independence between latent variables' variational distributions $q(z)$]

Therefore we have:

$$ELBO = E_{-j}\big[ log~p(x, z_j, z_{-j}) \big] - E_{q_j}[q_j] + const $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.