# Variational Inference: Computation of ELBO and CAVI algorithm

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed:

$$\mu_k \sim \mathcal{N}(0, \sigma^2) \\ c_i \sim Categorical(1/K, ... 1/K) \\ x_i|c_i, \mu \sim \mathcal{N}(c_i^{T}\mu, 1)$$

The joint density is modeled as follows:

$$p(x, c, \mu) = p(\mu)\prod_{i=1}^{n}p(c_i)p(x_i|c_i, \mu)$$

Using the mean-field approximation as,

$$q(\mu, c) = \prod_{k=1}^{K}q(\mu_k; m_k, s_k^{2}) \prod_{i=1}^{n}q(c_i;\varphi_i)$$

the authors arrive to the ELBO,

$$ELBO(\textbf{m}, \textbf{s}^2, \varphi) = \sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k, s_k^{2}] + \\ + \sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i] + \mathbb{E}[\log p(x_i|c_i, \mu); \textbf{m}, \textbf{s}^{2}, \varphi_i]) + \\ - \sum_{i=1}^{n}\mathbb{E}[\log q(c_i;\varphi_i)] - \sum_{k=1}^{K}\mathbb{E}[\log q(\mu_k; m_k, s_k^{2})]$$

I am kind of lost in how to compute the ELBO. E.g., the first term is the prior on $$\mu_k$$, which is a zero-mean Gaussian. Then I would say that term is zero. Am I right? In the second term, $$\sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i]$$, should it be $$\log (K)$$? Can someone give me a hint how to compute this equation?

Besides this, the paper goes on presenting the update algorithm on page 14. The update equation for the latent variables $$\varphi_i$$ is:

For $$i=1....n$$

$$\varphi_{ik} \propto \texttt{exp}\{\mathbb{E}[\mu_k; m_k, s_k^{2}]x_i - \mathbb{E}[\mu_k^{2};m_k,s_k^{2}]/2\}$$

Again, $$\mathbb{E}[\cdot]$$ is computed w.r.t. $$q(\cdot)$$, and, assuming that $$\mu_k$$ is a Gaussian distribution centered here at $$m_k$$, the first term should be simply $$\texttt{exp}\{m_k x_i\}$$ ? the second term just $$\texttt{exp}\{(s_k^{2} + m_k^{2})/2\}$$ ?

Help in understanding these expressions would be very appreciated! Thanks!

• I have recently found a python implementation of the example described in the paper here: am207.github.io/2017/wiki/VI.html. The ELBO is avoided and instead the updated variational parameters $m_k$ and $s_k^{2}$ are checked for convergence. Still, I would be glad to check whether I have fully understood the equation I cited in the original question. Oct 2 '18 at 11:26

Ok, I believe I got some feeling about what , e.g., the first term of the ELBO might be:

$$\sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k;s_k^{2}] \sum_{k=1}^{K}\mathbb{E}[-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\mathbb{E}[\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mathbb{E}[\mu_k^{2};m_k;s_k^{2}]}{2\sigma^{2}} = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}} = -\frac{K}{2}\log(\frac{1}{2\pi\sigma^2})-\sum_{k=1}^{K}\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}}$$

which is a function of the variational parameters and hence can be computed.

This might be a bit late but I would like to state what I have understood.

For your first question, the answer is no, the term is $$E[log(p(\mu))]$$. Here the expectation is over the distribution $$q_l(z_l)$$. Hence $$E_{q(\mu_k)}[log(p(\mu_k))] = \int q(\mu_k; m_k, s_k)log(p(\mu_k)) d\mu_k$$ which is not zero(it might be but we have to calculate it). It would have been zero for $$E_{p(\mu_k)}[\mu_k]$$.

For second term, it should be -log(K) simply because we have assumed the prior $$p(c_i)=1/k$$ i.e we have considered it to be uniform in the beginning. So our prior tells us that $$x_i$$ could be from any cluster with same probabilty which is not that useful :) .

About $$\phi_{ik}$$, I guess they have misprinted it. It says that it uses equation 18, hence the expectation is $$E_{-k}$$ and not $$E$$.