Obtaining VAE reconstruction probability

Question

How does one calculate the reconstruction probability? Let's look at the keras example code from here. Is the reconstruction probability the output of a specific layer, or is it to be calculated somehow?

According to the cited paper, the reconstruction probability is the "probability of the data being generated from a given latent variable drawn from the approximate posterior distribution". [2]

[2] Variational Autoencoder based Anomaly Detection using Reconstruction Probability - Jinwon An, Sungzoon Cho

Answer 1

Actually, the author of the original paper (Variational Autoencoder based Anomaly Detection using Reconstruction Probability - Jinwon An, Sungzoon Cho) abused the vocabulary. Also note that the author were not consistent when defining the reconstruction probability. In page 8 they defined it as the expectation of the log likelihood under the latent variable. In page 9 they estimated the expectation of the probability.

As a result some papers refer to An and Cho with different understanding of the term reconstruction probability. As an example here is a published work using the definition of page 9 (http://proceedings.mlr.press/v95/guo18a.html) and another paper using the definition of page 8 (https://arxiv.org/pdf/1802.03903.pdf)

Also, note that when you read reconstruction probability you would think that the value is between 0 and 1. But it is not the case. For instance, in case you use a continuous variable for your variational autoencoder output you would use the pdf of that variable. But the density has only the requirement to integrate to 1. Meaning that your "reconstruction probabily" can be very high in certain cases.

At the end, the definition you use depend on your problem. In my case working with distribution over sequences it wasn't stable to use the definition of page 9. Using the log likelihood made more sense. The actual expectation could be approximated using Monte Carlo after having sampled some latent vectors corresponding to the data point of interest.

Answer 2

Look at this tensorflow implementation. https://github.com/Michedev/VAE_anomaly_detection/blob/master/VAE.py

The author 1. build two dense nets to obtain $\vec \mu$ and $\vec \sigma$ at the last layer of the decoder (line 94); 2. use multivariate_normal.pdf function to compute reconstruction probability (line 161).

Answer 3

The reconstruction probability is $f(x|\vec \mu, \vec \sigma$) where $f(\cdot | \vec \mu, \vec\sigma)$ is the density of a normal distribution with mean $\vec \mu$ and diagonal covariance $\vec \sigma$.

$\vec \mu$ and $\vec \sigma$ are indeed the outputs of the encoder part of the VAE. Please refer to an implementation here. The reconstruction probability is calculated using the " reconstructed_probability" function.