I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some trouble with the coordinate ascent update for the mixture assignment probabilities.

Keeping in line with the paper's notation, we have that

$\mu \sim \text{N}(0, \sigma^2)$

$c \sim \text{Multinomial}(1, [\frac{1}{K}, \ldots \frac{1}{K}]) $

$x|\mu, c \sim \text{N}(c^T\mu, 1)$

The paper posits a multinomial as the variational distribution of the mixture assignments $c_i$. The optimal update for the mixture assignments is then given by

$q^{*}(c_i) \propto exp(\mathbb{E}_{-j}[log\, p(\mathbf{x}, \mathbf{c}, \mu)])$

$\propto exp(log \, p(c_i) + \mathbb{E}[log \, p(x_i|c_i, \mu)])$

$\propto exp(\mathbb{E}[log\, p(x_i | c_i, \mu)]) $

Where the expectation is taken with respect to $q(\mu)$. The inner term is equal to

$\mathbb{E}[log\, p(x_i | c_i, \mu)] = \sum_{k=1}^{k} c_{ik}\mathbb{E}[log\, p(x_i|\mu_k)]$

$\propto \sum_{k} c_{ik} \mathbb{E}[-\frac{(x_i-\mu_{k})^2}{2\sigma^2}]$

$\propto \sum_{k} c_{ik}(\mathbb{E}[\mu_k]x_i - \mathbb{E}[\mu_k^2]/2)$

From this, the authors conclude that the update for the multinomial variational parameter is

$P_q(c_{ik} = 1) = \psi_{ik} \propto exp(\mathbb{E}[\mu_k]x_i - \mathbb{E}[\mu_k^2]/2)$

I am struggling with the last step of the derivation. The second to last step had an extra $c_{ik}$ term that seemed to just disappear, and I can't find out how they were able to cancel it out.


1 Answer 1


$c_i$ is an "indicator vector". This means that $c_{ik}$ is 1 only when $x_i$ belongs to cluster $k$, and 0 otherwise. Thus when we are considering $\psi_{ik}$, we can directly evaluate the expression $$\exp\left(\mathbb{E}[\mu_k]x_i - \frac{\mathbb{E}[\mu_k^2]}2\right)$$ for that particular $k$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.