I noticed that in most of the implementations of a variational autoencoder with gaussian posterior, the variance of the gaussian is not learned during training. The decoder usually outputs only the means of the Gaussian, whereas the covariance matrix is usually fixed to be the identity matrix. Moreover, at inference time we never sample from the gaussian, instead, we assume that the mean of the distribution is the reconstructed sample.
Why we do not parametrize the covariance matrix? How would this affect the training?
