I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization:

\begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2) \\ c_i & \sim categorical(\frac{1}{K}) \\ x_i~\vert ~c_i, \mathbf{\mu} & \sim \mathcal{N}(\mu_{c_i}, 1) \end{align}

Then, mean field variational family is introduced for latent variables $\mu$ and $c$ in the following form:

\begin{align} q(\mathbf{\mu}, \mathbf{c}) &= \prod_{k} q(\mu_k; m_k, s_k^2) \prod_{i}q(c_i; \phi_i) \end{align}

My question is that why we need to introduce variational distribution for $\mu$'s? In the probabilistic model, we have already assumed that they come from a normal distribution with 0 mean and $\sigma^2$ variance, which is a hyperparameter.

Without that, we can still derive the optimal solutions for $\phi$'s first, and then $\mu$'s by taking the partial derivatives from ELBO, right? I am a bit confused.


So I emailed Prof. David Blei and he responded me as follows:

the prior and posterior of mu are different. the model's distribution is the prior. the variational distribution seeks to approximate its posterior.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.