Defining ELBO in Variational Inference with 3 random variables

Question

I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $x,y,z$, we would want to define the ELBO in two conditions, when only $z$ is latent (Eq.6), when both $y,z$ are latent (Eq.7). The first case is:

Eq.6: $-\mathcal{L}(x,y) = \mathop{\mathbb{E}}_{q_{ \phi}(z|x,y)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(z|x,y)]$

and the second case is:

Eq.7: $\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)]$

The difference between the two is that in the second one $y$ is assumed to be latent, while in the first equation $y$ is observed. Now, based on the above two, they define :

$\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)] \\= \sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$

which I really can't derive from the two equations. I tried expanding RHS ($\sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$) to recover the LHS of Eq.7. But, no hope so far ...

Answer 1

I am going to merge the generative distribution for readability: $\log p_{\theta}(x,y,z) = \log p_{\theta}(x|yz) + \log p(y) + \log p(z)$.

Start by assuming the following decomposition of the variational distribution $q_{\phi}(y,z|x) = q_{\phi}(z|x,y)q_{\phi}(y|x)$ and let $z$ be continuous and $y$ discrete, we can write out the expectation.

\begin{align} \log p(x) &\geq \mathbb{E}_{q_{\phi}(y,z|x)} [\log p_{\theta}(x, y, z) - \log q_{\phi}(y, z|x)]\\ &= \sum_y \int_z q_{\phi}(y|x) q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(y|x) - \log q_{\phi}(z|x,y)]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] - \log q_{\phi}(y|x) \right ]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] \right ] - \sum_y q_{\phi}(y|x)\log q_{\phi}(y|x)\\ &= \sum_{y} q_{\phi}(y|x) (- \mathcal{L(x,y)}) + \mathcal{H}(q_{\phi}(y|x)) = - \mathcal{U}(x) \end{align}