# How to convert VAE L2 Reconstruction Loss to a Log Likelihood

I have been puzzled trying to convert MSE to Log Likelihood in VAEs. Relevant Questions:

What is bits per dimension (bits/dim) exactly (in pixel CNN papers)?

Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model?

Relevant Discussion: Reddit: [Discussion] Calculation of bits/dims

In the paper: Masked Autoregressive Flow for Density Estimation

They provide a formula from going from "Pixel space" to logit space, but I don't understand the logic behind it.

They normalize the pixel values and then multiply by some hyper parameter that is chosen arbitrarily.

They then derive this formula:

For which it is not clear if $$x_i$$ is an image in the dataset/batch or a pixel of image $$x_i$$ (most likely the later one but still unsure)

For which $$x_i$$ is a pixel value of image x.

But it is not clear what is p(x) for my VAE trained on MSE.

They normalize the pixel values and then multiply by some hyper parameter that is chosen arbitrarily.

Actually, they're scaling and shifting an interval [0,255] to be [$$\lambda,1-\lambda$$], then applying logit to. Logit can't handle 0 or 1 as you know: $$\mathrm{logit}(x)=\ln\frac x {1-x}$$, so they have to put a floor and a ceiling on its inputs. They floor $$\lambda$$ is arbitrary in some sense.

They also "dequantize" the pixel values, by adding random noise, so the values become, sort of, continuous.

For which it is not clear if $$x_i$$ is an image in the dataset/batch or a pixel of image $$x_i$$ (most likely the later one but still unsure)

They denote $$x$$ the set of pixels of an image which has D pixels, but in logit space. So $$x_i$$ would be a pixel in logit space. The result of the formula is a density bits per pixel, which they get from $$p(x)$$ - density in logit space.

• so is $p(x)$ calculated directly from the $x=$ relationship? if so what is it exactly in practice? Commented Feb 18, 2020 at 16:09
• in the paper in this particular equation it's the density directly from the model. remember they logit transform the 256 bit pixels, z space, into x space which is "almost" $x\in [-\infty,\infty]$. the model fits to this space, and produces p(x) density, that's what they use in this equation Commented Feb 18, 2020 at 16:16
• Sorry it is still not clear to me where to calculate the $p(x)$ from in the VAE context. Commented Feb 18, 2020 at 17:06
• Would you please explain what is "Density"? It'd be great if you can provide some actual example. Commented Apr 16, 2022 at 6:48