A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It's also complicated when using some reconstruction loss $R$ as a log-likelihood for some complex data-type. Balancing the KL term and $R$ is itself a challenge, and this issue is seemingly built into the use of $p(x)$. Theis et al, A Note on the Evaluation of Generative Models note some of these difficulties.

However, we can easily measure the likelihood of the generative model sampling the encoding of given data point $x$: it's just $p_z( E(x) )$, where $E$ is the encoder, and $p_z$ is the latent prior. However, this does not tell us if the output is good, i.e. if $\hat{x}=D(x)$ is close to $x$. In other words, we might be able to sample the encoded latent form of $x$, but this does not mean the reconstruction will be good. But we can use $R$ for that! In other words, if there is a good chance of sampling $E(x)$ under the prior AND the reconstruction performance is good, then (seemingly) we can say that the generative model is good.

So why not report the two values: the average $R(x)$ and $\log p_z(E(x))$ over the held-out set? Are there any papers that do this?

  • $\begingroup$ i can't really tell where $p_z(E(x))$ came from. it isn't the probability that $x$ is drawn from the generative model $\endgroup$ – shimao Mar 24 '19 at 0:35
  • $\begingroup$ @shimao that was badly worded, sorry. I've edited it. $\endgroup$ – user3658307 Mar 24 '19 at 0:39

Well, $p_z(E(x))$ doesn't seem to be the best metric as it can be maximized with with a encoder network which disregards the standard normal prior and always outputs 0 (or some very small values close to 0, so that the decoder can do its job).

More broadly, when comparing VAEs to other generative models (RBMs, PixelRNN, PixelCNN, GLOW, NICE, many GAN variants, etc) which don't have any autoencoder structure and don't have any concept of "reconstruction loss" or "likelihood of the encoding of a data point", it doesn't make sense to use either of those measures. You can only really rely on 1. the subjective quality of the generated samples, and 2. likelihood of the model (although GANs don't allow this, and with VAEs it can be hard)

  • $\begingroup$ Thanks for your answer! Regarding para 1, that's why I suggested also reporting $R$. (Presumably $R$ would be poor under posterior collapse. The MI between x and its encoding would be small.) Anecdotally, I've found that the average $\log p(E(x))$ seems generally related to the quality of VAE samples I get (when $R$ is good). Regarding para 2, I suppose you're right. It's what I often see in papers, though some also report $R$. I was hoping for literature on other measures though (specifically these). For instance, in this work they define another metric. $\endgroup$ – user3658307 Mar 24 '19 at 1:56
  • $\begingroup$ @user3658307 well encoding a posterior of always ~0 wouldn't be posterior collapse -- just imagine a non-variational autoencoder, but where the weights are scaled so that the latent space has tiny value. in the VAE case, it would have a huge KL loss and poor samples, but good reconstructions. $\endgroup$ – shimao Mar 24 '19 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.