Semi-supervised objective function VAE In Kingma's paper on Semi-supervised learning https://arxiv.org/pdf/1406.5298.pdf, we are shown equations for the ELBO for the semisupervised case, however I am having a hard trying to derive the math from the vanilla VAE objective. Specifically, Eq.6 in the paper refers to the situation where we have labels (supervised) and Eq.7 refers to no labels (unsupervised) and 'y' is treated as a latent variable. Can someone provide a derivation for these two objectives or provide some pointers on how to derive these?
Eq.6 for labelled data is:
$log p_{θ}(x,y) \ge E_{q_{\phi} (z|x,y}) [ log p_\theta(x|y,z) + log p_{θ}(y) + log p(z) - log q_\phi(z|x,y)]  $
Eq.7 for unlabeled data is :
$log p_{θ}(x) \ge E_{q_{\phi} (y,z|x}) [ log p_\theta(x|y,z) + log p_{θ}(y) + log p(z) - log q_\phi(y,z|x)]  $
=
$ \sum_y q_\phi (y|x) (-L(x,y)) + H(q_\phi(y|x)) $
Thanks!
 A: I'll derive equation 6. Hopefully this will illustrate the principles you can apply yourself for equation 7.
Equation 6 deals with the case where we observe both the input $x$ and class label $y$. Using Bayes' rule, the posterior distribution over latent variables $z$ is:
$$p_\theta(z \mid x,y) =
\frac{p_\theta(x \mid y,z) p_\theta(y) p(z)}{ p_\theta(x,y)}$$
where $\theta$ is a parameter vector. $p(z)$ is considered fixed, so is not indexed by $\theta$. Note that the above expression reflects the paper's assumption that the prior factorizes as $p_\theta(y,z) = p_\theta(y) p(z)$.
A variational distribution $q_\phi(z \mid x,y)$ is defined in equation 4. This is meant to approximate the true posterior above. The quality of this approximation can be measured using the KL divergence from the variational distribution to the true posterior:
$$D_{KL} \Big( q_\phi(z \mid x,y) \parallel p_\theta(z \mid x,y) \Big) \ = \
E_{q_\phi(z \mid x,y)} \Big[
  \log q_\phi(z \mid x,y)  - \log p_\theta(z \mid x,y)
\Big]$$
I'll use $KL$ as shorthand for the KL divergence above. Plug in the expression for the true posterior:
$$KL \ = \
E_{q_\phi(z \mid x,y)} \Big[
  \log q_\phi(z \mid x,y) 
  - \log p_\theta(x \mid y,z)
  - \log p_\theta(y)
  -\log p(z)
  + \log p_\theta(x,y)
\Big]$$
$\log p_\theta(x,y)$ is the log evidence. This can be moved out of the expectation because it doesn't depend on $z$:
$$KL \ = \
E_{q_\phi(z \mid x,y)} \Big[
  \log q_\phi(z \mid x,y) 
  - \log p_\theta(x \mid y,z)
  - \log p_\theta(y)
  -\log p(z)
\Big] + \log p_\theta(x,y)$$
Move the expectation and KL divergence to the same side, and note that $-E[V] = E[-V]$. This gives:
$$\log p_\theta(x,y) =
KL + E_{q_\phi(z \mid x,y)} \Big[
  -\log q_\phi(z \mid x,y) 
  + \log p_\theta(x \mid y,z)
  + \log p_\theta(y)
  + \log p(z)
\Big]$$
KL divergence is always nonnegative, so $KL \ge 0$. Therefore:
$$\log p_\theta(x,y) \ge
E_{q_\phi(z \mid x,y)} \Big[
  -\log q_\phi(z \mid x,y) 
  + \log p_\theta(x \mid y,z)
  + \log p_\theta(y)
  + \log p(z)
\Big]$$
This is equation 6.
A: Posting a derivation using the Jensen's inequality.
$ p(x,y) = \int_z p(x,y,z) = \int_z p(x,y,z) \cdot \dfrac{q(z/x,y)}{q(z/x,y)} = E_{q(z/x,y)} \dfrac{p(x,y,z)}{q(z/x,y)}$
Applying the log on both sides of the equation
$logp(x,y) = log E_{q(z/x,y)} \dfrac{p(x,y,z)}{q(z/x,y)} $ and now applying the Jensen inequality we get
$logp(x,y) \ge  E_{q(z/x,y)} [logp(x,y,z) - log(q(z/x,y)] $
Now expand the joint distribution according to the factorization assumptions in the paper (Eq.2) and we get Eq. 6
$logp(x,y) \ge  E_{q(z/x,y)} [logp(y) + logp(z) + logp(x/y,z) - logq(z/x,y)]$
