2
$\begingroup$

I'm deeply failing to understand the first step in the ELBO derivation in VAEs.

When asking my questions I'll also try to clearly state my assumptions and perhaps some of them are wrong to begin with:

In VAEs our encoder learns to approximate the conditional distribution $q_\phi (z | x)$. This distribution (which is forced to be a gaussian if I'm understanding correctly) represents the distribution of possible $z$ values under a given data point $x$.

  • Question 1:

In the ELBO derivation, we have a decoder parameterized by $\theta$, and we suggest that.

$log p_\theta(x) = \mathbb{E}_{q_\phi(z|x)}[logp_\theta(x)]$

Im not sure if my question makes sense but I don't understand why the above equation is valid. Why is the expectation of $x$ with respect to the conditional distribution $q_\phi (z | x)$, which again means under a specific $x$ value yields the distribution of all $x$ values over all possible data points

I understand that given we know $q_\phi (z | x)$, or again, the distribution of $z$ values under a known $x$ value, we can feed them to our decoder and get a distribution of $x$ values, but shouldn't this be the distribution of $x$ under a certain z value (the one sampled from the conditional distribution outputted by the encoder)? and not over all possible x values?

  • Question 2:

By forcing the encoder to output $\mu$ and $\Sigma$ values for the conditional distribution $q_\phi(z | x)$ we are basically defining it as a gaussian.

I am unsure however why does this mean that the prior distribtion $p_\theta(z)$ is necessarily a gaussian as well. doesn't this depend on the distribution of $x$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I don't where your ELBO term comes from. But if I am not wrong the ELBO loss is given as (taken from here)

$ \begin{equation} L_{\text{VAE}} = - \mathbb{E}_{q_{\phi}(z|x)}{\log{p_{\theta}(x|z)}} + D_{KL}(q_{\phi}(z|x)||p_{\theta}(z)) \end{equation} $

Question 1 : From this notation, one understands that the expectation is a conditional expectation conditioned on the value of X=x. To me it makes sense in this equation but I am not sure about where your term comes from.

Question 2: We choose the prior distribution to be standard normal distribution. We can choose whatever distribution we want in theory. The motivation to choose a standard normal distribution as prior is because it is easy to work with them and usually we choose the distribution to be the same family as posterior distribution (more on this). Check this for examples of VAE which use other distribution family for posterior and prior

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you! Regarding my 2nd question: im not sure about the following. I understand that the posterior is our choice and doesnt have to be a gaussian, my question is, given that the posterior is chosen to be a gaussiam, does it necessitate mathematically that the prior is also a gaussian? It doesnt make sense to me if it does, but it seems as if in the paper they treat it as this is the case (or my understanding is poor) $\endgroup$
    – DrPrItay
    Commented Oct 16 at 11:44
  • 1
    $\begingroup$ @DrPrItay, choosing gaussian posterior does not necessitate that prior should also be gaussian. You can choose whatever distribution you want as the prior. They simply choose gaussian as the prior because they can arrive at closed form solutions for the KL divergence term between posterior and prior. $\endgroup$
    – the-nihilist-ninja
    Commented Oct 16 at 11:53
  • $\begingroup$ This is exactly my question. It doesnt seem that there is a "choice" regarding the prior but rather the posterior.... what am I missing here, the encoder outouts the $\mu$ and $\sigma$ of the posterior distribution and not the prior.... $\endgroup$
    – DrPrItay
    Commented Oct 16 at 12:01
  • $\begingroup$ @DrPrItay If by choice you mean a learnable distribution, then yeah we don't have a choice with regards to prior distribution. But what I mean by choice is the ability to choose the distribution family and the parameters for it...... thus you can choose for instance a beta distribution with parameters (alpha=2, beta=1). I am not sure if there are variations of VAE where the prior too is learnable. $\endgroup$
    – the-nihilist-ninja
    Commented Oct 16 at 12:12
  • $\begingroup$ Thank you! I understand that we can choose what distribution we want, what I was confused (and still am) is that it seems that in most cases you choose the posterior and not the prior, and I was wondering if particularly, under the case which we choose that the posterior is a gaussian (by learning its' mu / sigma) does that necessitate that the entire $p_{\theta}(z)$ and not just $q_{\phi}(z|x)$ distribution is also a gaussian (or a multivariate gaussian for that matter) $\endgroup$
    – DrPrItay
    Commented Oct 16 at 12:21
0
$\begingroup$

The first part is true because $x$ is constant in the expectation, the integral is taken with regards to $z$.

$$\log p_\theta(x) = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x)]=\int{\log p_\theta(x)}q_\phi(z|x)dz$$

$p_\theta(z)$ is a choice, it could be anything we wanted. Gaussian is chosen based on nice properties, but often you'll see other distributions (von Mises-Fisher, or Student's t distributions being somewhat common). The amortized approximate posterior $q_\phi (z | x)$ also doesn't have to match the prior at all, that is also a choice.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you! Regarding my 2nd question: im not sure about the following. I understand that the posterior is our choice and doesnt have to be a gaussian, my question is, given that the posterior is chosen to be a gaussiam, does it necessitate mathematically that the prior is also a gaussian? It doesnt make sense to me if it does, but it seems as if in the paper they treat it as this is the case (or my understanding is poor) $\endgroup$
    – DrPrItay
    Commented Oct 16 at 11:45
  • $\begingroup$ @DrPrItay Categorically no, you can have whatever distributions you want for approximate posterior and prior. It's just that Gaussian leads to closed-form expressions, but it didn't have to be the case $\endgroup$
    – Firebug
    Commented Oct 16 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.