3
$\begingroup$

In a Variational Autoencoder we say that the encoder and decoder networks model p(z | x) and p(x | z) respectively as seen in the image below: VAE

My question is -- is the output of the decoding layer a sample from the approximate p(x|z) itself, or a set of probability values corresponding to the probability of each pixel for some decoded $\mathbf{z}$? If it is a set of probability values for each dimension of $\mathbf{x}$, how is it that we consider the input to also be a set of probability values for each pixel? Just because of the normalization between (0,1)? Could someone convince me of that if true?

$\endgroup$
1
$\begingroup$

The "decoder" outputs a sample from $p_\theta(x \mid z)$, e.g. "reconstructed image" in your example. It doesn't output a probability at all. It's just a coincidence that it happens to output pixel values in $[0,1]$ in this example; they could equally be in $[0,255]$ and nothing about the model would change (except you'd have to use a different activation on the final layer).

$\endgroup$
4
  • $\begingroup$ Why then is binary cross entropy sometimes used for the reconstruction loss? And see the discussion on page 17 of arxiv.org/pdf/1606.05908.pdf which suggests that one is estimating the probability $\endgroup$
    – Einstein
    Oct 19 '18 at 2:56
  • 1
    $\begingroup$ From the link: "Admittedly this is not quite what the VAE framework prescribes but works well in practice". That's a way you can work with MNIST but it's not quite a standard VAE, and it probably doesn't make sense eg on color images. $\endgroup$
    – Danica
    Oct 19 '18 at 3:00
  • $\begingroup$ Right ok - are you saying the BCE only applies if data to be reconstructed and output are normalized 0,1? Say for example my data was not image pixel values at all but just a 2d array of values between (0,infinity). In that reconstruction, BCE obv wouldn't apply because out of (0,1) range ? Then, say we are using MSE reconstruction, we would say that our decoder is not the probability density function p(x|z) but rather a tool that draws samples X that follow P(X | z). Thanks very much for your insights!! $\endgroup$
    – Einstein
    Oct 19 '18 at 3:05
  • $\begingroup$ Right, exactly. Even more obvious is if outputs could be negative; it's just not a sensible loss to apply in that case. $\endgroup$
    – Danica
    Oct 19 '18 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.