1
$\begingroup$

In Kingma's paper on Semi-supervised learning https://arxiv.org/pdf/1406.5298.pdf, we are shown equations for the ELBO for the semisupervised case, however I am having a hard trying to derive the math from the vanilla VAE objective. Specifically, Eq.6 in the paper refers to the situation where we have labels (supervised) and Eq.7 refers to no labels (unsupervised) and 'y' is treated as a latent variable. Can someone provide a derivation for these two objectives or provide some pointers on how to derive these?

Eq.6 for labelled data is: $log p_{θ}(x,y) \ge E_{q_{\phi} (z|x,y}) [ log p_\theta(x|y,z) + log p_{θ}(y) + log p(z) - log q_\phi(z|x,y)] $

Eq.7 for unlabeled data is : $log p_{θ}(x) \ge E_{q_{\phi} (y,z|x}) [ log p_\theta(x|y,z) + log p_{θ}(y) + log p(z) - log q_\phi(y,z|x)] $

= $ \sum_y q_\phi (y|x) (-L(x,y)) + H(q_\phi(y|x)) $

Thanks!

$\endgroup$
2
  • 1
    $\begingroup$ Please add the necessary details into body of your question. $\endgroup$
    – Tim
    Oct 11, 2019 at 19:41
  • 1
    $\begingroup$ Equations added to the body of the question as requested $\endgroup$
    – Rebelzane
    Oct 11, 2019 at 23:52

2 Answers 2

1
$\begingroup$

I'll derive equation 6. Hopefully this will illustrate the principles you can apply yourself for equation 7.

Equation 6 deals with the case where we observe both the input $x$ and class label $y$. Using Bayes' rule, the posterior distribution over latent variables $z$ is:

$$p_\theta(z \mid x,y) = \frac{p_\theta(x \mid y,z) p_\theta(y) p(z)}{ p_\theta(x,y)}$$

where $\theta$ is a parameter vector. $p(z)$ is considered fixed, so is not indexed by $\theta$. Note that the above expression reflects the paper's assumption that the prior factorizes as $p_\theta(y,z) = p_\theta(y) p(z)$.

A variational distribution $q_\phi(z \mid x,y)$ is defined in equation 4. This is meant to approximate the true posterior above. The quality of this approximation can be measured using the KL divergence from the variational distribution to the true posterior:

$$D_{KL} \Big( q_\phi(z \mid x,y) \parallel p_\theta(z \mid x,y) \Big) \ = \ E_{q_\phi(z \mid x,y)} \Big[ \log q_\phi(z \mid x,y) - \log p_\theta(z \mid x,y) \Big]$$

I'll use $KL$ as shorthand for the KL divergence above. Plug in the expression for the true posterior:

$$KL \ = \ E_{q_\phi(z \mid x,y)} \Big[ \log q_\phi(z \mid x,y) - \log p_\theta(x \mid y,z) - \log p_\theta(y) -\log p(z) + \log p_\theta(x,y) \Big]$$

$\log p_\theta(x,y)$ is the log evidence. This can be moved out of the expectation because it doesn't depend on $z$:

$$KL \ = \ E_{q_\phi(z \mid x,y)} \Big[ \log q_\phi(z \mid x,y) - \log p_\theta(x \mid y,z) - \log p_\theta(y) -\log p(z) \Big] + \log p_\theta(x,y)$$

Move the expectation and KL divergence to the same side, and note that $-E[V] = E[-V]$. This gives:

$$\log p_\theta(x,y) = KL + E_{q_\phi(z \mid x,y)} \Big[ -\log q_\phi(z \mid x,y) + \log p_\theta(x \mid y,z) + \log p_\theta(y) + \log p(z) \Big]$$

KL divergence is always nonnegative, so $KL \ge 0$. Therefore:

$$\log p_\theta(x,y) \ge E_{q_\phi(z \mid x,y)} \Big[ -\log q_\phi(z \mid x,y) + \log p_\theta(x \mid y,z) + \log p_\theta(y) + \log p(z) \Big]$$

This is equation 6.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer! I have derived a version using the Jensen inquality and posted it below, if it is not too much trouble would you be able to take a look at it? I derived Eq.7 along similar lines but I am not sure the approach is correct. However, I have accepted your answer since you posted it first! $\endgroup$
    – Rebelzane
    Oct 13, 2019 at 1:00
  • $\begingroup$ @Rebelzane Glad to help. Starting from either KL divergence or Jensen's inequality are both valid ways to derive the ELBO. I'll take a look. $\endgroup$
    – user20160
    Oct 13, 2019 at 1:15
1
$\begingroup$

Posting a derivation using the Jensen's inequality.

$ p(x,y) = \int_z p(x,y,z) = \int_z p(x,y,z) \cdot \dfrac{q(z/x,y)}{q(z/x,y)} = E_{q(z/x,y)} \dfrac{p(x,y,z)}{q(z/x,y)}$

Applying the log on both sides of the equation

$logp(x,y) = log E_{q(z/x,y)} \dfrac{p(x,y,z)}{q(z/x,y)} $ and now applying the Jensen inequality we get

$logp(x,y) \ge E_{q(z/x,y)} [logp(x,y,z) - log(q(z/x,y)] $

Now expand the joint distribution according to the factorization assumptions in the paper (Eq.2) and we get Eq. 6

$logp(x,y) \ge E_{q(z/x,y)} [logp(y) + logp(z) + logp(x/y,z) - logq(z/x,y)]$

$\endgroup$
2
  • $\begingroup$ Looks correct (+1). Just a minor point: The first line should contain an integral rather than a sum, since $z$ is continuous. $\endgroup$
    – user20160
    Oct 13, 2019 at 1:20
  • $\begingroup$ Thank you! Answer edited to change the summation to an integral for continuous z. $\endgroup$
    – Rebelzane
    Oct 13, 2019 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.