0
$\begingroup$

In variational bayes, distributions in the variational family $\mathcal{Q}$ are denoted $q_\phi$ and are used to approximate the posterior $p_\theta(z|x)$. However, I've seen both notations $q_\phi(z|x)$ and $q_\phi(z)$. What's the difference and how are they each used?

My interpretation was that $q_\phi(z)$ and $q_\phi(z|x)$ are a prior and posterior on $\mathcal{Q}$. Is this the right interpretation? It doesn't make sense that we should need a prior for $q_\phi$ since it is being optimized as an approximation to $p_\theta(z|x)$.

$\endgroup$
2
$\begingroup$

"My interpretation was that $q_{\phi}(𝑧)$ and $q_{\phi}\left(z | x \right)$ are a prior and posterior on $\mathcal{Q}$" is not correct at all. $\mathcal{Q}$ is the variational family defining the distribution space. $q_{\phi}(𝑧)$ and $q_{\phi}\left(z | x \right)$ live in $\mathcal{Q}$ and used to approximate the prior/posterior defined for a data distribution.

Both notations are used actually. $q_{\phi}(𝑧)$ actually stems from the ELBO derivation(You can check these lecture notes from CMU), i.e. the KL divergence definition.

For VAE's, we specifically denote $q_{\phi}\left(z | x \right)$ because the encoder tries to learn an amortized posterior distribution, which instead of optimizing a set of free parameters unlike in the lecture notes that I've shared sbove, we can propose parameterized function(like a neural network) that maps from observation space to the parameters of the approximate posterior distribution.

In fact, there are advanced VI techniques that allow implicit priors for VAE's like Semi-Implicit Variational Inference (SIVI).

TL;DR, try to stick with $q_{\phi}\left(z | x \right)$ for VAE posteriors and always make sure to understand which distribution you are trying to approximate.

Hope this helps.

$\endgroup$
4
  • $\begingroup$ Thanks for your helpful answer! So why is amortized inference called that? Is it the fact that we aren't computing $q_\phi(z_i)$ for each individual datapoint but rather defining a global function and then letting each datapoint contribute? ($\leftarrow$ is this right?) $\endgroup$ – 900edges Mar 23 at 12:50
  • $\begingroup$ Ah I did some reading online, will edit your answer for my own reference if that's okay with you $\endgroup$ – 900edges Mar 23 at 12:52
  • 1
    $\begingroup$ @900edges There's nothing wrong with writing your own answer to the question. $\endgroup$ – Sycorax Mar 23 at 13:49
  • $\begingroup$ @Sycorax I want to accept their answer, just wanted to fill out a detail! $\endgroup$ – 900edges Mar 23 at 13:52
1
$\begingroup$

Note: the point of Amortized VI is this:

Traditional VI defines the latent variable $z_i$ for each individual observation, and then optimizes them all jointly. This is computationally expensive to have to re-run the inference procedure for every datapoint. Instead, one can define a parameterized function, i.e. $q_\phi(z|x)$ sending $x \rightarrow z$. Now the parameters to update are the parameters of the function rather than the distributions of every $z_i$. For a new observation, we only have to pass it through the network rather than update everything.

$\endgroup$
1
  • 1
    $\begingroup$ Good follow-up. Thank you @900edges $\endgroup$ – Emir Ceyani Mar 28 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.