I'm practicing with VAEs for generative purpose, and from what I understood we need the latent variable $z$ to approximate a distribution, usually the standard normal $N(0, I)$.

In any example I could find we use $z \sim N(\mu(h;\theta), \Sigma(h;\theta))$ where $\theta$ are the optimization parameters and $h$ is the encoder's output.

It is very apparent to me that $\mu$ should depend from $h$ because that's the way the information gets from the encoder to the latent space, but I cannot understand the choice of having $\Sigma = \Sigma(h;\theta)$.

In all my optimization attempts the realization of $\Sigma$ consistently just gets near $I$ as much as possible, hence I guess I could simply use $\Sigma = I$ or maybe $\Sigma = \Sigma(\theta)$ with no dependency on $h$.

This would reduce the number of parameters improving speed. Why don't we actually do that?


1 Answer 1


To my knowledge, the output of the encoder $ (\mu(x),\Sigma(x))=f_E(x) $ is usually the only thing used to parametrize the resulting encoder distribution, i.e. $q(z|x) = \mathcal{N}(z|\mu(x),\Sigma(x))$. Now, let me give a conceptual, then a practical reason why for your question.

Theoretically speaking, our encoder distribution is a variational approximation to the true posterior. In other words, the variational posterior is being fit to the true posterior. By removing the $\Sigma$, you are weakening this fit. Generally, using a normal approximation is already a pretty weak approximation (that's why techniques like Variational Inference with Normalizing Flows exist), so you don't really want to weaken it even more!

Practically speaking, you often don't actually want $\Sigma(x)=I$. It really depends on the situation. Keep in mind that you are optimizing the ELBO, not the true marginal likelihood, which makes it seems like $\Sigma(x)=I$ is ideal (since it minimizes $ \mathfrak{D}_\text{KL}[q(z|x)\mid\mid p(z)] $, where $p(z)=\mathcal{N}(0,I)$). However, it would be better to be minimizing $ \mathfrak{D}_\text{KL}[q(z)\mid\mid p(z)] $ where $q(z) = \int q(z|x) p_\text{empirical}(x) dx$, but this is intractable. Clearly this shows that $q$ should be adaptive based on the input. Otherwise, the marginal posterior may not match the prior as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.