How to preserve statistical properties of vectors after applying an autoencoder? I have trained an autoencoder that minimizes mean squared error (MSE) between input vectors and reconstructed vectors. By looking at data before and after transformation, I found out that the mean values across each dimension are almost the same, but the variance values have changed.
If an autoencoder tries to reconstruct the original vectors, shouldn't the mean and variance values remain almost the same before and after transformation?
To maintain mean and variance values, do I need to use a variational autoencoder (VAE) instead? Do denoising autoencoders work better than regular autoencoders for this purpose?
 A: It makes sense that the variance would have changed, and without seeing the data, I'm sure that what you're seeing more specifically is that the variance of the output vectors is smaller than that of the input vectors, correct?
The reason why this is happening is that because the autoencoder cannot perfectly reconstruct all input samples, it has to decide where/how to accept some lossyness. The easiest way to do this is allow for degraded accuracy on outlier data points by outputting values closer to the population mean, since these will (on average) result in a smaller MSE than would be achieved by outputting more extreme values.
One possible way to fix this problem would be to add a variance term to the autoencoder's loss function. Barring that, a variational autoencoder is indeed a good alternative.
I do not expect that a denoising autoencoder would help in your situation, in fact I would expect it would make the variance discrepancy even more pronounced, since denoising partially works by dragging "extreme" values closer to the mean.
