Timeline for Why sampling from the posterior is a good estimate for the Likelihood but sampling from the prior is bad?
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 9 at 1:48 | vote | accept | rando | ||
| Jul 7 at 20:19 | comment | added | Ruben van Bergen | (To relate this back to the faces example: it's (typically) not the case that we have such a strong prior that despite a face clearly having green eyes, we'll get a $q(z|x)$ that puts low probability on green eyes and high probability on brown eyes. At most, our approximate posterior will be a very slight biased towards brown eyes. That is, of course, unless you're evaluating your model in an out-of-distribution setting, with test data sampled from a markedly different distribution from what the VAE was trained on.) | |
| Jul 7 at 20:10 | comment | added | Ruben van Bergen | By "strong prior", I meant something extreme enough to overrule the likelihood. That's not normally the case in VAEs. If you sample $z^*\sim q(z|x)$ and then project this back through the decoder, you'll usually get something back that is close to the input $x$ (e.g. an image similar to the handwritten digit you put in). So this tells you that the likelihood dominates the inference, and most any $z$ sampled from $q(z|x)$ will lead to high $p(x|z)$. | |
| Jul 7 at 19:54 | comment | added | rando | Thank you. You said that $p(x|z)$ would typically be high for values of $z$ where $p(z|x)$ is high in the absence of strong prior. That is not the case of VAE since the prior is zero-mean diagonal-covariance Gaussian. Although $z$ is highly dimensional, we have restricted the values of $z$ by introducing the KL divergence. Your example of faces is spot on, but under the condition that there is no regularization on the space of $z$. Do you agree? | |
| Jul 7 at 19:12 | history | answered | Ruben van Bergen | CC BY-SA 4.0 |