Variational autoencoder: Why reconstruction term is same to square loss?

Question

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as:

$L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$

While in the code, the second term $\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$ is actually achieved by: binary_crossentropy(x, x_output), where x and x_output is input and output of autoencoder respectively.

My question is why are the losses of input and output is equivelant to $\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$?

Answer 1

For regular Autoencoders, you start from an input, $x$ and encode it to obtain your latent variable (or code), $z$, using some function that satisfy: $z=f(x)$. After getting the latent variable, you aim to reconstruct the input using some other function $\hat{x}=g(f(x))$. The reconstruction loss is yet another function $L(x,\hat{x})$ that you use to back-propagate and update $f$ and $g$.

For Variational Autoencoders, you still interpret the latent variables, $z$, as your code. Hence, $p(x|z)$ serves as a probabilistic decoder, since given a code $z$, it produces a distribution over the possible values of $x$. It thus "makes sense" that the term $\log p_{\theta}(x|z)$ is somehow connected to reconstruction error.

Both encoder and decoder are deterministic functions. Since $p(x|z)$ is such function that maps $z$ into $\hat{x}$ , you can think of this expression as $p(x|\hat{x})$. When you assume (as they assumed in the paper if I understood it correctly) that this distribution have a Gaussian form: $$ \log P(x|\hat{x}) \sim \log e^{-|x-\hat{x}|^2} \sim (x-\hat{x})^2 $$

The last expression is proportional to the reconstruction error in regular autoencoders.

Answer 2

$p(x|z) = p(x|x_\text{out}) = x \log x_\text{out} + (1-x) \log (1-x_\text{out})$ because $x_\text{out}$ is a deterministic function of $z$ and the model parameters.

Answer 3

To answer this one needs to see page 4 eq. 7 of this and text below it:

In our experiments we found that the number of samples L per datapoint can be set to 1 as long as the minibatch size M was large enough, e.g. M = 100.

So stochastic nature of Monte-Carlo sampling from standard normal distribution (sampling $L$ samples of $z^{(i, l)}$ from $\mu^{(i)}$ and $\sigma^{(i)}$) can be neglected with large enough minibatch size. Hence summing over $l$ of $L$ samples turnes into single sample. So what is left is only network output that is parameter $p_{out}^{(i)}$ of Bernoulli distribution (that is deterministic fuction of $z^{(i)}$). And hence:

$$ \log P(x^{(i)}|z^{(i)}) = \log P(x|p_{out}) \\= \log P_{p_{out}}(X=x) \\= \log \left(p_{out}^x (1-p_{out})^{1-x}\right) \\= x \log p_{out} + {(1-x)} \log \left(1-p_{out}\right), $$

where $x^{(i)}$ is assumed to be from Bernoulli distribution ($\in \left\{0, 1\right\}$) and third equation was exactly The probability mass function of Bernoulli distribution.

As to why this works when x is not from Bernoulli but instead from bounded continuous like ($\in \left[0, 1\right]$) I don't know but may be it's relatred to "Dark Knowledge" refered here.