I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance vector will suit the task.
However, I noticed that even with MNIST digit data, once the network is trained to reconstruct the digits very well, the encoder gives a larger variance (vector norm) for actual digit images than for a complete noise image. Why is that? Why is the predicted variance higher for 'clean' data than it is for a random pixel image?