# Defining ELBO in Variational Inference with 3 random variables

I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $$x,y,z$$, we would want to define the ELBO in two conditions, when only $$z$$ is latent (Eq.6), when both $$y,z$$ are latent (Eq.7). The first case is:

Eq.6: $$-\mathcal{L}(x,y) = \mathop{\mathbb{E}}_{q_{ \phi}(z|x,y)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(z|x,y)]$$

and the second case is:

Eq.7: $$\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)]$$

The difference between the two is that in the second one $$y$$ is assumed to be latent, while in the first equation $$y$$ is observed. Now, based on the above two, they define :

$$\mathop{\mathbb{E}}_{q_{ \phi}(y,z|x)}[\log P_\theta(x|y,z)+\log P_\theta(y)+\log P(z) - \log q_{ \phi}(y,z|x)] \\= \sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$$

which I really can't derive from the two equations. I tried expanding RHS ($$\sum_y q_{\phi}(y|x)(-\mathcal{L}(x,y)) + \mathcal{H}(q_{\phi}(y|x))$$) to recover the LHS of Eq.7. But, no hope so far ...

## 1 Answer

I am going to merge the generative distribution for readability: $$\log p_{\theta}(x,y,z) = \log p_{\theta}(x|yz) + \log p(y) + \log p(z)$$.

Start by assuming the following decomposition of the variational distribution $$q_{\phi}(y,z|x) = q_{\phi}(z|x,y)q_{\phi}(y|x)$$ and let $$z$$ be continuous and $$y$$ discrete, we can write out the expectation.

\begin{align} \log p(x) &\geq \mathbb{E}_{q_{\phi}(y,z|x)} [\log p_{\theta}(x, y, z) - \log q_{\phi}(y, z|x)]\\ &= \sum_y \int_z q_{\phi}(y|x) q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(y|x) - \log q_{\phi}(z|x,y)]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] - \log q_{\phi}(y|x) \right ]\\ &= \sum_y q_{\phi}(y|x) \left [ \int_z q_{\phi}(z|x,y) [\log p_{\theta}(x,y,z) - \log q_{\phi}(z|x,y)] \right ] - \sum_y q_{\phi}(y|x)\log q_{\phi}(y|x)\\ &= \sum_{y} q_{\phi}(y|x) (- \mathcal{L(x,y)}) + \mathcal{H}(q_{\phi}(y|x)) = - \mathcal{U}(x) \end{align}

• Nice, but it seems you have assumed two decompositions for $q(y,z|x)$ in $\mathbb{E}_{q(y,z|x)} [\log p(x, y, z) - \log q(y, z|x)]$, once as $q(y|x)q(z|x)$ to write the expectation, and once as $q(z|x,y)q(y|x)$ to expand the $\log q(y, z|x)$ where each corresponds to a particular graphical model over $x,y,z$ variables. Oct 8, 2018 at 14:20
• Yes, I have edited my answer to fix this issue. Oct 8, 2018 at 15:36