Is the assumption of a diagonal covariance matrix on the latent space in a variational autoencoder in any way restrictive?

Question

The covariance matrix in an autoencoder is assumed to be diagonal. And, I see it mentioned in good places that this is a fairly restrictive assumption. To quote

However, in order to simplify the computation and reduce the number of parameters, we make the additional assumption that our approximation of p(z|x), q_x(z), is a multidimensional Gaussian distribution with diagonal covariance matrix (variables independence assumption). With this assumption, h(x) is simply the vector of the diagonal elements of the covariance matrix and has then the same size as g(x). However, we reduce this way the family of distributions we consider for variational inference and, so, the approximation of p(z|x) obtained can be less accurate.

Question: How is the assumption of a diagonal covariance matrix a restrictive assumption when you can express any multivariate gaussian (with any covariance matrix) through a linear transformation on multivariate gaussian with diagonal covariance matrix? That linear transformation would the inverse of the whitening matrix and could be learned as a part of the decoder

Answer 1

You seem to be confusing the prior with the posterior. While it's true that you can use a simple prior, like a standard Gaussian, and still have a decoder neural network that maps it to an arbitrarily complex distribution, the situation is different for the posterior. A more complicated encoder induces a more complex true posterior, which is generally not Gaussian, let alone a diagonal Gaussian. If your encoder cannot produce a variational distribution that approximates the true posterior well enough given a particular decoder, you won't be able to learn that decoder through variational inference. This is because the ELBO punishes distance between true posterior and variational distribution. Therefore, having a restricted class of variational distributions effectively limits the type of encoder models you can learn.