5
$\begingroup$

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as:

$L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$

While in the code, the second term $\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$ is actually achieved by: binary_crossentropy(x, x_output), where x and x_output is input and output of autoencoder respectively.

My question is why are the losses of input and output is equivelant to $\frac{1}{L}\sum_{l=1}^L \log p_\theta(x^i|z^{i,l})$?

$\endgroup$
  • $\begingroup$ Your last sentence doesn't actually match the title of the question. If you have a look at Algorithm 1, they assume L = 1. In case the probabilistic decoder is a Bernoulli distribution, p(x|z) is clearly represented by the binary cross entropy. $\endgroup$ – lenhhoxung Jan 4 '19 at 0:53
5
$\begingroup$

For regular Autoencoders, you start from an input, $x$ and encode it to obtain your latent variable (or code), $z$, using some function that satisfy: $z=f(x)$. After getting the latent variable, you aim to reconstruct the input using some other function $\hat{x}=g(f(x))$. The reconstruction loss is yet another function $L(x,\hat{x})$ that you use to back-propagate and update $f$ and $g$.

For Variational Autoencoders, you still interpret the latent variables, $z$, as your code. Hence, $p(x|z)$ serves as a probabilistic decoder, since given a code $z$, it produces a distribution over the possible values of $x$. It thus "makes sense" that the term $\log p_{\theta}(x|z)$ is somehow connected to reconstruction error.

Both encoder and decoder are deterministic functions. Since $p(x|z)$ is such function that maps $z$ into $\hat{x}$ , you can think of this expression as $p(x|\hat{x})$. When you assume (as they assumed in the paper if I understood it correctly) that this distribution have a Gaussian form: $$ \log P(x|\hat{x}) \sim \log e^{-|x-\hat{x}|^2} \sim (x-\hat{x})^2 $$

The last expression is proportional to the reconstruction error in regular autoencoders.

$\endgroup$
1
$\begingroup$

$p(x|z) = p(x|x_\text{out}) = x \log x_\text{out} + (1-x) \log (1-x_\text{out})$ because $x_\text{out}$ is a deterministic function of $z$ and the model parameters.

$\endgroup$
  • $\begingroup$ I don't quite follow your second equality. Would you mind explaining? $\endgroup$ – Jon Deaton Dec 27 '18 at 9:58
  • $\begingroup$ @JonDeaton yes -- it is just the expression for cross-entropy between $x$ and $x_\text{out}$ -- you can replace it with mse loss if appropriate. $\endgroup$ – shimao Dec 27 '18 at 16:07
0
$\begingroup$

To answer this one needs to see page 4 eq. 7 of this and text below it:

In our experiments we found that the number of samples L per datapoint can be set to 1 as long as the minibatch size M was large enough, e.g. M = 100.

So stochastic nature of Monte-Carlo sampling from standard normal distribution (sampling $L$ samples of $z^{(i, l)}$ from $\mu^{(i)}$ and $\sigma^{(i)}$) can be neglected with large enough minibatch size. Hence summing over $l$ of $L$ samples turnes into single sample. So what is left is only network output that is parameter $p_{out}^{(i)}$ of Bernoulli distribution (that is deterministic fuction of $z^{(i)}$). And hence:

$$ \log P(x^{(i)}|z^{(i)}) = \log P(x|p_{out}) \\= \log P_{p_{out}}(X=x) \\= \log \left(p_{out}^x (1-p_{out})^{1-x}\right) \\= x \log p_{out} + {(1-x)} \log \left(1-p_{out}\right), $$

where $x^{(i)}$ is assumed to be from Bernoulli distribution ($\in \left\{0, 1\right\}$) and third equation was exactly The probability mass function of Bernoulli distribution.

As to why this works when x is not from Bernoulli but instead from bounded continuous like ($\in \left[0, 1\right]$) I don't know but may be it's relatred to "Dark Knowledge" refered here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.