Variational Inference: Computation of ELBO and CAVI algorithm

Question

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed:

$\mu_k \sim \mathcal{N}(0, \sigma^2) \\ c_i \sim Categorical(1/K, ... 1/K) \\ x_i|c_i, \mu \sim \mathcal{N}(c_i^{T}\mu, 1)$

The joint density is modeled as follows:

$p(x, c, \mu) = p(\mu)\prod_{i=1}^{n}p(c_i)p(x_i|c_i, \mu)$

Using the mean-field approximation as,

$q(\mu, c) = \prod_{k=1}^{K}q(\mu_k; m_k, s_k^{2}) \prod_{i=1}^{n}q(c_i;\varphi_i)$

the authors arrive to the ELBO,

$ELBO(\textbf{m}, \textbf{s}^2, \varphi) = \sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k, s_k^{2}] + \\ + \sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i] + \mathbb{E}[\log p(x_i|c_i, \mu); \textbf{m}, \textbf{s}^{2}, \varphi_i]) + \\ - \sum_{i=1}^{n}\mathbb{E}[\log q(c_i;\varphi_i)] - \sum_{k=1}^{K}\mathbb{E}[\log q(\mu_k; m_k, s_k^{2})]$

I am kind of lost in how to compute the ELBO. E.g., the first term is the prior on $\mu_k$, which is a zero-mean Gaussian. Then I would say that term is zero. Am I right? In the second term, $\sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i]$, should it be $\log (K)$? Can someone give me a hint how to compute this equation?

Besides this, the paper goes on presenting the update algorithm on page 14. The update equation for the latent variables $\varphi_i$ is:

For $i=1....n$

$ \varphi_{ik} \propto \texttt{exp}\{\mathbb{E}[\mu_k; m_k, s_k^{2}]x_i - \mathbb{E}[\mu_k^{2};m_k,s_k^{2}]/2\}$

Again, $\mathbb{E}[\cdot]$ is computed w.r.t. $q(\cdot)$, and, assuming that $\mu_k$ is a Gaussian distribution centered here at $m_k$, the first term should be simply $\texttt{exp}\{m_k x_i\}$ ? the second term just $\texttt{exp}\{(s_k^{2} + m_k^{2})/2\}$ ?

Help in understanding these expressions would be very appreciated! Thanks!

Answer 1

Ok, I believe I got some feeling about what , e.g., the first term of the ELBO might be:

$\sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k;s_k^{2}] \sum_{k=1}^{K}\mathbb{E}[-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\mathbb{E}[\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mathbb{E}[\mu_k^{2};m_k;s_k^{2}]}{2\sigma^{2}} = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}} = -\frac{K}{2}\log(\frac{1}{2\pi\sigma^2})-\sum_{k=1}^{K}\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}}$

which is a function of the variational parameters and hence can be computed.

Answer 2

This might be a bit late but I would like to state what I have understood.

For your first question, the answer is no, the term is $E[log(p(\mu))]$. Here the expectation is over the distribution $q_l(z_l)$. Hence $E_{q(\mu_k)}[log(p(\mu_k))] = \int q(\mu_k; m_k, s_k)log(p(\mu_k)) d\mu_k$ which is not zero(it might be but we have to calculate it). It would have been zero for $E_{p(\mu_k)}[\mu_k]$.

For second term, it should be -log(K) simply because we have assumed the prior $p(c_i)=1/k$ i.e we have considered it to be uniform in the beginning. So our prior tells us that $x_i$ could be from any cluster with same probabilty which is not that useful :) .

About $\phi_{ik}$, I guess they have misprinted it. It says that it uses equation 18, hence the expectation is $E_{-k}$ and not $E$.