I try to understand this paper where they try to use a mixture of Gaussian as a prior, instead of the standard gaussian. There are several things unclear to me though:

  1. They say that they set $\pi_k = \frac{1}{K}$ and draw $z$ from Cat($\pi$). But later in equation 5 they parameterize $p_\beta(z_k = 1|x, w)$ by a neural network, and also condition it on the inputs. How does that fit together?
  2. Also in equation 5 they calculate a KL divergence. What is this KL divergence explicitly though? How to optimize it?
  3. They don't write how to optimize the $z$-prior term in equation 4. Is there a closed form solution similar to the $w$-prior term? Or how would one do that?
  1. Cat($\pi$) is the prior distribution on $z$. $p_\beta(z|x,w)$ is the variational approximation of the posterior of $z$.

  2. It's the KL between the variational approximation of the posterior of $x$ (technically it's just one factor of the mean field variational approximation) and the conditional prior of $x$. These are both normal distributions, so computing the KL is easy and differentiable.

  3. I believe $p(z)$ is just Cat($\pi$), so the KL can be written in closed form.

  • $\begingroup$ So 2. simply with the reparametrization trick? And for 3. what is the closed form of the KL? $\endgroup$ – Luca Thiede Nov 30 '18 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.