# Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as:

$$L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$$

While in the code, the second term $$\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$$ is actually achieved by: binary_crossentropy(x, x_output), where x and x_output is input and output of autoencoder respectively.

My question is why are the losses of input and output is equivelant to $$\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$$?

• Your last sentence doesn't actually match the title of the question. If you have a look at Algorithm 1, they assume L = 1. In case the probabilistic decoder is a Bernoulli distribution, p(x|z) is clearly represented by the binary cross entropy. – lenhhoxung Jan 4 '19 at 0:53

For regular Autoencoders, you start from an input, $x$ and encode it to obtain your latent variable (or code), $z$, using some function that satisfy: $z=f(x)$. After getting the latent variable, you aim to reconstruct the input using some other function $\hat{x}=g(f(x))$. The reconstruction loss is yet another function $L(x,\hat{x})$ that you use to back-propagate and update $f$ and $g$.

For Variational Autoencoders, you still interpret the latent variables, $z$, as your code. Hence, $p(x|z)$ serves as a probabilistic decoder, since given a code $z$, it produces a distribution over the possible values of $x$. It thus "makes sense" that the term $\log p_{\theta}(x|z)$ is somehow connected to reconstruction error.

Both encoder and decoder are deterministic functions. Since $p(x|z)$ is such function that maps $z$ into $\hat{x}$ , you can think of this expression as $p(x|\hat{x})$. When you assume (as they assumed in the paper if I understood it correctly) that this distribution have a Gaussian form: $$\log P(x|\hat{x}) \sim \log e^{-|x-\hat{x}|^2} \sim (x-\hat{x})^2$$

The last expression is proportional to the reconstruction error in regular autoencoders.

$p(x|z) = p(x|x_\text{out}) = x \log x_\text{out} + (1-x) \log (1-x_\text{out})$ because $x_\text{out}$ is a deterministic function of $z$ and the model parameters.

• I don't quite follow your second equality. Would you mind explaining? – Jon Deaton Dec 27 '18 at 9:58
• @JonDeaton yes -- it is just the expression for cross-entropy between $x$ and $x_\text{out}$ -- you can replace it with mse loss if appropriate. – shimao Dec 27 '18 at 16:07

To answer this one needs to see page 4 eq. 7 of this and text below it:

In our experiments we found that the number of samples L per datapoint can be set to 1 as long as the minibatch size M was large enough, e.g. M = 100.

So stochastic nature of Monte-Carlo sampling from standard normal distribution (sampling $$L$$ samples of $$z^{(i, l)}$$ from $$\mu^{(i)}$$ and $$\sigma^{(i)}$$) can be neglected with large enough minibatch size. Hence summing over $$l$$ of $$L$$ samples turnes into single sample. So what is left is only network output that is parameter $$p_{out}^{(i)}$$ of Bernoulli distribution (that is deterministic fuction of $$z^{(i)}$$). And hence:

$$\log P(x^{(i)}|z^{(i)}) = \log P(x|p_{out}) \\= \log P_{p_{out}}(X=x) \\= \log \left(p_{out}^x (1-p_{out})^{1-x}\right) \\= x \log p_{out} + {(1-x)} \log \left(1-p_{out}\right),$$

where $$x^{(i)}$$ is assumed to be from Bernoulli distribution ($$\in \left\{0, 1\right\}$$) and third equation was exactly The probability mass function of Bernoulli distribution.

As to why this works when x is not from Bernoulli but instead from bounded continuous like ($$\in \left[0, 1\right]$$) I don't know but may be it's relatred to "Dark Knowledge" refered here.