1
$\begingroup$

The original presentation of variational autoencoders, VAE assumes the mean $\mu$ and the sd $\sigma$ are functions of the input variable, say $x$. I am studying "Learning Structured Output Representation using Deep Conditional Generative Models" by Sohn et al. and came across the lines

The prior of the latent variables $z$ is modulated by the input $x$ in our formulation; however, the constraint can be easily relaxed to make the latent variables statistically independent of input variables, i.e., $p_\theta (z|x) = p_\theta (z)$

and the only justification to this statement is citation of "Semi-supervised Learning with Deep Generative Models" by Kingma et al., which I have not not yet found to be supportive of the claim. My questions:

  • Could anyone justify or shed lights to this claim?
  • Most VAE tutorials talk about the generative power of it using only the decoder and samples from the latent representation (aka, throw away the encoder), is this really the case since $\mu = \mu(x)$ and $\sigma = \sigma(x)$?
  • An answer in How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian? points to the fact that there is a misconception as people often take $\mu$ and $\sigma$ as constant learnable parameters. Any support, justification or reference to this statement?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

  1. By construction you reduce the KL divergence between the conditional and the prior, so your $p(z|x)$ is almost your prior (hopefully), thus you can think it as a distribution by itself

  2. the fact that you calculate $\mu$ and $\sigma$ based on the input is only allowing you to train the model, but the task is not "reconstruction", but "sampling"... in other words, high probability density area in your original distribution should be mapped to high probability density area in your latent distribution (in other words, you are not interested in "what will come out, but just that it follows the original distribution)

  3. $\mu$ and $\sigma$ (if you mean the encoder output) are for sure not constants and for sure not something that you aim to learn, as you are not interested in them, if not for training purposes (ie having a smooth latent space, thanks to sampling)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could you explain what you mean in point 1)? $p_\theta (z/x)$ and $p_\theta (z)$ are distributions by definition. For 2), I understand the idea of VAE but looking to understand variations I've seen in practice. Thanks for point 3), so how do you sample from the learned distribution after training? You need the training samples? $\endgroup$
    – Toonia
    Commented Oct 13, 2022 at 0:38
  • $\begingroup$ @Toonia 1) you train your model minimizing the distance between the conditional wrt the prior, thus after training your conditional should be similar to that, which means that the “|x” is almost irrelevant… 2&3) with VAE u don’t care what you are sampling, as long as it seems to come from the original distribution… at inference, you sample from a isotropic gaussian and you plug it as latent representation, and you use your decoder to get the transformation from the original distribution (this is why u dont need the encoder) $\endgroup$
    – Alberto
    Commented Oct 14, 2022 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.