I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $x,y,z$, we would want to define the ELBO in two conditions, when only $z$ is latent (Eq.6), when both $y,z$ are latent (Eq.7). The first case is:

Eq.6: $-\mathcal{L}(x,y) = \mathop{\mathbb{E}}_{q_{ \phi}(z|x,y)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(z|x,y)]$

and the second case is:

Eq.7: $\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)]$

The difference between the two is that in the second one $y$ is assumed to be latent, while in the first equation $y$ is observed. Now, based on the above two, they define :

$\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)] \\= \sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$

which I really can't derive from the two equations. I tried expanding RHS ($\sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$) to recover the LHS of Eq.7. But, no hope so far ...


1 Answer 1


I am going to merge the generative distribution for readability: $\log p_{\theta}(x,y,z) = \log p_{\theta}(x|yz) + \log p(y) + \log p(z)$.

Start by assuming the following decomposition of the variational distribution $q_{\phi}(y,z|x) = q_{\phi}(z|x,y)q_{\phi}(y|x)$ and let $z$ be continuous and $y$ discrete, we can write out the expectation.

\begin{align} \log p(x) &\geq \mathbb{E}_{q_{\phi}(y,z|x)} [\log p_{\theta}(x, y, z) - \log q_{\phi}(y, z|x)]\\ &= \sum_y \int_z q_{\phi}(y|x) q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(y|x) - \log q_{\phi}(z|x,y)]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] - \log q_{\phi}(y|x) \right ]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] \right ] - \sum_y q_{\phi}(y|x)\log q_{\phi}(y|x)\\ &= \sum_{y} q_{\phi}(y|x) (- \mathcal{L(x,y)}) + \mathcal{H}(q_{\phi}(y|x)) = - \mathcal{U}(x) \end{align}

  • $\begingroup$ Nice, but it seems you have assumed two decompositions for $q(y,z|x)$ in $\mathbb{E}_{q(y,z|x)} [\log p(x, y, z) - \log q(y, z|x)]$, once as $q(y|x)q(z|x)$ to write the expectation, and once as $q(z|x,y)q(y|x)$ to expand the $\log q(y, z|x)$ where each corresponds to a particular graphical model over $x,y,z$ variables. $\endgroup$ Commented Oct 8, 2018 at 14:20
  • 1
    $\begingroup$ Yes, I have edited my answer to fix this issue. $\endgroup$
    – wohlert
    Commented Oct 8, 2018 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.