0
$\begingroup$

In the simple case of mixture of gaussians(with known variance), we have 2 latent variables $\mu$ and $z$. In the vaiational auto-encoder, we assume that the model is infinite mixture of gaussians. If we assume that the variance is known then we have 2 latent variables $\mu$ and $z$. the encoder the responsible for calculating the variational distribution of $z$ (its mean and variance). and the decoder is responsible for calculating the variational distribution of the $\mu$ (it's mean and variance)
So the variance $\sigma$ that the generative network is outputting Is NOT the variance of $x$
My questions is
Is my viewpoint correct?
I read many articles on the internet that say $P(x|z, w) = \mathcal{N}(\mu(z, w),\sigma(z, w))$ where $\mu(.)$ and $\sigma(.)$ are the generative network with parameters $w$. Isn't that wrong?

PS: I know that training and results are the same in both cases

$\endgroup$
1
$\begingroup$

In a GMM, $z$ is a latent variable, but the $\mu$'s are a model parameter.

You seem to be trying to draw an analogy between GMMs and VAEs -- it's true you can think of a VAE as an infinite mixture model, where each component has $\mu = f(z; \theta)$ where $f$ is the decoder. Here $\theta$, not $\mu$, are the model parameters.

and the decoder is responsible for calculating the variational distribution of the $\mu$

No -- $\mu$ isn't a latent variable, so there's no need for any variational approximation for it.

Yes, $x|z$ has distribution $\mathcal{N}(f(z;\theta))$ -- this is simply how the VAE model is defined.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I read before that we can treat GMM as a fully bayesian model and treat $\mu$ as a latent variable and calculate it using variational inference or we can just treat $\mu$ as a parameter and calculate it using EM. Maybe VAE is a variational EM not just variational inference(meaning that we don't treat all the unknown parameters $\mu$ and $z$ as latent variables). Am I right? $\endgroup$ – floyd Sep 12 '19 at 7:19
  • 1
    $\begingroup$ i'm not familiar with this "fully bayesian" form of GMM, so I can't comment on that. A VAE is not trained with EM. It is very much a variational inference method. $z$ is definitely a latent variable. $\endgroup$ – shimao Sep 12 '19 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.