1
$\begingroup$

I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance vector will suit the task.

However, I noticed that even with MNIST digit data, once the network is trained to reconstruct the digits very well, the encoder gives a larger variance (vector norm) for actual digit images than for a complete noise image. Why is that? Why is the predicted variance higher for 'clean' data than it is for a random pixel image?

$\endgroup$
1
$\begingroup$

A VAE models a distribution as

$$\log P(x) = \log \int P(x|z)P(z) dz \geq E_{z \sim q}[\log P(x|z)] - \text{KL}(q(z)||p(z))$$

As you can see, there is no direct relationship between the variance of $q$ and the probability the model assigns to a given datapoint $x$.

In fact decreasing the variance on $q$ could increase the KL term, resulting in lower log likelihood. Anyway I suggest using the ELBO or a monte carlo estimate of the integral to determine how much probability the model assigns to a particular image.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.