Variational Autoencoders (VAEs) are based on the concept of Variational Inference (VI) and use two Neural Networks similar to Vanilla Autoencoders (AEs) for function approximation. I understood the derivation of the Evidence-Lower-Bound (ELBO) and the role of the two ELBO terms that make up the objective for training a VAE:
- Expectation of P(x|z) with z ~ Q(z|x) -> "reconstruction loss"
- KL-Divergence of Q(z|x) and P(z) -> "regularization term"
The second regularization term is very clear to me, as it basically forces the latent representation to be distributed like the prior on z.
However, the first one is not as intuitive to me. Let's assume we speak about the standard VAE that assumes Gaussians as Q(z|x) and P(x|z), where the encoder produces means and variances for the latent dimensions of z and the decoder produces the means for P. For this specific VAE, the first loss term is equal to the MSE loss between the predicted mean of the decoder and the input x, isn't it?
My intuition ends here: Why can we use this reconstruction loss to judge the pixel-wise differences between the input x and the decoded latent sample z? I mean, x gets encoded and specifies some latent distribution over z's. Now we sample from it to get a specific z. But this z does not have to be meant to be the latent code for the input, doesn't it? So the decoded version of z does not have to show the same image content? Let's say we are using face images. The input could be male. The sampled z could be producing a female person? So using MSE between these two seems to be wrong?