# Is the assumption of a diagonal covariance matrix on the latent space in a variational autoencoder in any way restrictive?

The covariance matrix in an autoencoder is assumed to be diagonal. And, I see it mentioned in good places that this is a fairly restrictive assumption. To quote

However, in order to simplify the computation and reduce the number of parameters, we make the additional assumption that our approximation of p(z|x), q_x(z), is a multidimensional Gaussian distribution with diagonal covariance matrix (variables independence assumption). With this assumption, h(x) is simply the vector of the diagonal elements of the covariance matrix and has then the same size as g(x). However, we reduce this way the family of distributions we consider for variational inference and, so, the approximation of p(z|x) obtained can be less accurate.

Question: How is the assumption of a diagonal covariance matrix a restrictive assumption when you can express any multivariate gaussian (with any covariance matrix) through a linear transformation on multivariate gaussian with diagonal covariance matrix? That linear transformation would the inverse of the whitening matrix and could be learned as a part of the decoder