# On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $$p(x)$$. This is difficult and often one can only get a lower bound. It's also complicated when using some reconstruction loss $$R$$ as a log-likelihood for some complex data-type. Balancing the KL term and $$R$$ is itself a challenge, and this issue is seemingly built into the use of $$p(x)$$. Theis et al, A Note on the Evaluation of Generative Models note some of these difficulties.

However, we can easily measure the likelihood of the generative model sampling the encoding of given data point $$x$$: it's just $$p_z( E(x) )$$, where $$E$$ is the encoder, and $$p_z$$ is the latent prior. However, this does not tell us if the output is good, i.e. if $$\hat{x}=D(x)$$ is close to $$x$$. In other words, we might be able to sample the encoded latent form of $$x$$, but this does not mean the reconstruction will be good. But we can use $$R$$ for that! In other words, if there is a good chance of sampling $$E(x)$$ under the prior AND the reconstruction performance is good, then (seemingly) we can say that the generative model is good.

So why not report the two values: the average $$R(x)$$ and $$\log p_z(E(x))$$ over the held-out set? Are there any papers that do this?

• i can't really tell where $p_z(E(x))$ came from. it isn't the probability that $x$ is drawn from the generative model – shimao Mar 24 '19 at 0:35
• @shimao that was badly worded, sorry. I've edited it. – user3658307 Mar 24 '19 at 0:39

Well, $$p_z(E(x))$$ doesn't seem to be the best metric as it can be maximized with with a encoder network which disregards the standard normal prior and always outputs 0 (or some very small values close to 0, so that the decoder can do its job).
• Thanks for your answer! Regarding para 1, that's why I suggested also reporting $R$. (Presumably $R$ would be poor under posterior collapse. The MI between x and its encoding would be small.) Anecdotally, I've found that the average $\log p(E(x))$ seems generally related to the quality of VAE samples I get (when $R$ is good). Regarding para 2, I suppose you're right. It's what I often see in papers, though some also report $R$. I was hoping for literature on other measures though (specifically these). For instance, in this work they define another metric. – user3658307 Mar 24 '19 at 1:56