Why does the encoder from a variational autoencoder map to a vector of means and a vector of standard deviations? Why does it not instead map to a vector of means and a covariance matrix?

Is it because we want our latent vectors to have zero covariance between components?

  • $\begingroup$ One reason is that it's much quicker to draw samples from a Gaussian with a diagonal covariance matrix. $\endgroup$
    – πr8
    Jan 23, 2019 at 0:21

3 Answers 3


The diagonal covariance matrix is an explicit statement about the kind of latent representation the researcher wants the model to learn: a representation that can be modeled as independent Gaussians.

Additionally, @Firebug points out in comments that a symmetric, PD matrix can be diagonalized without any loss of information. In other words, for some symmetric, PD matrix $A$, we can write $A=PDP^\top$ for $D$ some diagonal matrix and $P$ can be chosen to be orthonormal. This retains the same information in the sense that $A$ is rotated to have orthogonal coordinates.

Purely from the perspective of abstraction, there is no reason you must be limited to learning a latent representation that is composed of independent Gaussians. However, the computational side seems challenging.

The standard VAE encoder for a single sample produces latent parameters $(\mu, \sigma)$ its input. Then it uses the re-parameterization trick to draw random samples from that distribution. There are $d$ elements in each of $\mu$ and $\sigma$ so the total number of latent parameters is $2d$.

An alternative model that includes a covariance matrix would need some method to produce a covariance matrix, so the output of the encoder is $(\mu, \Sigma)$.

If your latent space has dimension $d$, you're doing inference on each of the $d$ elements of $\mu$ and each of the $\frac{d(d+1)}{2}$ elements of $\Sigma$ (because $\Sigma$ is symmetric by definition), for a total of $\frac{d(d+3)}{2}$ elements. Any time you have more than 1 latent dimension, the covariance matrix model will have more latent parameters to learn compared to the diagonal model.

Furthermore, the multivariate normal distribution requires that $\Sigma$ be positive definite, so we must somehow guarantee that, for each sample, we generate a PD matrix. (Using an alternative strategy, such as factorizing into standard deviations and correlation matrix $\Omega$, i.e. $\Sigma = (\sigma I) \Omega (\sigma I)$, will increase the number of effective parameters without solving the PD problem, since now we must guarantee that $\Omega$ is PD.)

Additionally, we must also be able to backprop that procedure so that the encoder weights can be updated. This may or may not be possible, depending on the strategy used to generate $\Sigma$ and draw a deviate from the multivariate Gaussian.

These three issues -- more parameters, assuring differentiability, positive definiteness -- are challenging.

If you're contemplating undertaking research to overcome these challenges, that's great! But one must ask, why is this a good model? What problems does it solve which are not solved by the diagonal Gaussian VAE model, or an alternative non-Gaussian VAE model (e.g. a Dirichlet VAE)?

  • 1
    $\begingroup$ I think that one more issue arises: if $\Sigma$ is PD, the you could diagonalize it without loss of information. Or, in other words, having a full-$\Sigma$ isn't any better than a diagonal one. $\endgroup$
    – Firebug
    Sep 8, 2020 at 20:16
  • $\begingroup$ I've had some interesting results on this, I might add an answer later $\endgroup$
    – Firebug
    Oct 21, 2020 at 12:05
  • $\begingroup$ @Firebug I'll look forward to reading it. In particular, I wonder what explanatory power you're gaining in exchange for the added complexity of estimating a covariance matrix. $\endgroup$
    – Sycorax
    Oct 21, 2020 at 13:15
  • $\begingroup$ I think this has been touched upon before, regarding non-realistic posterior specification. But I didn't really expect it to work tbh. $\endgroup$
    – Firebug
    Oct 21, 2020 at 13:27
  • $\begingroup$ I would add that diagonalizability doesn't mean diagonal matrices are equally expressive as full matrices as each latent distribution could still have different orientations. I can imagine there are data where this might increase performance if objects share basic identity but differ in orientation. A SPD can be easily parameterized through low rank-updates or explicit (low-rank) factorization, which all have little extra computational cost with auto-differentiation. Explicit factorization also makes sampling a breeze. Check the STAN docs for an example of a Cholesky parameterization. $\endgroup$ Jan 21 at 23:33

I'm working on something like this now.

Let's say the researcher wants a full covariance latent variable $Z ~ N(mu, \Sigma)$. Like the multiplication trick from VAE, we can have a matrix multiplication trick. Still sample from a high dimensional unit Gaussian, $e \sim N(0, I)$.

There is this property of linear transformations of high D Gaussians: https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations#hid2. If we want the transformed covariance to be $\Sigma$, we have to hit $e$ with the matrix square root of the proposal covariance $\Sigma^{1/2}e \sim N(0, \Sigma)$.

Someone above pointed out that $\Sigma$ must be symmetric positive definite. This can be modeled as a sum of rank 1 matrices, formed as the outer product of each component vector. This is a sum of quadratic forms, so we know it will at least be positive semidefinite. To get positive definite, we have to choose a sufficient number of component vectors. By CLT (> 30 components), the off diagonal entries will be zero mean Gaussian, while the main diagonal remains positive. The number of components needed in practice might depend on other factors.

I tried this with MNIST and got decent results. There might be a way to enforce PD by parametrizing $\Sigma$ with Inverse Wishart prior, but that also requires one of the parameters to be a symmetric PD matrix.

  • $\begingroup$ Very nice post! Wouldn't it be possible to directly learn the spectral components of $\Sigma^{1/2}$, e.g., $\Sigma^{1/2} = \frac{1}{N} \sum_{i=1}^{N} g_i(X) g_i(X)^{T} $? $\endgroup$
    – max0r
    Jun 24, 2021 at 11:08

A variational autoencoder is based on variational inference. The zero covariance is an assumption, mean-field variational family, which makes optimization easier, since the latent variables are independent.

Check chapter 2.3 of this paper: https://arxiv.org/abs/1601.00670

They also briefly dive into other families:

One way to expand the family is to add dependencies between the variables (Saul and Jordan, 1996; Barber and Wiegerinck, 1999); this is called structured variational inference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.