1
$\begingroup$

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed:

$\mu_k \sim \mathcal{N}(0, \sigma^2) \\ c_i \sim Categorical(1/K, ... 1/K) \\ x_i|c_i, \mu \sim \mathcal{N}(c_i^{T}\mu, 1)$

The joint density is modeled as follows:

$p(x, c, \mu) = p(\mu)\prod_{i=1}^{n}p(c_i)p(x_i|c_i, \mu)$

Using the mean-field approximation as,

$q(\mu, c) = \prod_{k=1}^{K}q(\mu_k; m_k, s_k^{2}) \prod_{i=1}^{n}q(c_i;\varphi_i)$

the authors arrive to the ELBO,

$ELBO(\textbf{m}, \textbf{s}^2, \varphi) = \sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k, s_k^{2}] + \\ + \sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i] + \mathbb{E}[\log p(x_i|c_i, \mu); \textbf{m}, \textbf{s}^{2}, \varphi_i]) + \\ - \sum_{i=1}^{n}\mathbb{E}[\log q(c_i;\varphi_i)] - \sum_{k=1}^{K}\mathbb{E}[\log q(\mu_k; m_k, s_k^{2})]$

I am kind of lost in how to compute the ELBO. E.g., the first term is the prior on $\mu_k$, which is a zero-mean Gaussian. Then I would say that term is zero. Am I right? In the second term, $\sum_{i=1}^{n}(\mathbb{E}[\log p(c_i);\varphi_i]$, should it be $\log (K)$? Can someone give me a hint how to compute this equation?

Besides this, the paper goes on presenting the update algorithm on page 14. The update equation for the latent variables $\varphi_i$ is:

For $i=1....n$

$ \varphi_{ik} \propto \texttt{exp}\{\mathbb{E}[\mu_k; m_k, s_k^{2}]x_i - \mathbb{E}[\mu_k^{2};m_k,s_k^{2}]/2\}$

Again, $\mathbb{E}[\cdot]$ is computed w.r.t. $q(\cdot)$, and, assuming that $\mu_k$ is a Gaussian distribution centered here at $m_k$, the first term should be simply $\texttt{exp}\{m_k x_i\}$ ? the second term just $\texttt{exp}\{(s_k^{2} + m_k^{2})/2\}$ ?

Help in understanding these expressions would be very appreciated! Thanks!

$\endgroup$
  • $\begingroup$ I have recently found a python implementation of the example described in the paper here: am207.github.io/2017/wiki/VI.html. The ELBO is avoided and instead the updated variational parameters $m_k$ and $s_k^{2}$ are checked for convergence. Still, I would be glad to check whether I have fully understood the equation I cited in the original question. $\endgroup$ – mgbacher Oct 2 '18 at 11:26
1
$\begingroup$

Ok, I believe I got some feeling about what , e.g., the first term of the ELBO might be:

$\sum_{k=1}^{K}\mathbb{E}[\log p(\mu_k);m_k;s_k^{2}] \sum_{k=1}^{K}\mathbb{E}[-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\mathbb{E}[\frac{\mu_k^{2}}{2\sigma^{2}};m_k;s_k^{2}] = \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{\mathbb{E}[\mu_k^{2};m_k;s_k^{2}]}{2\sigma^{2}} = \\ \sum_{k=1}^{K}-\frac{1}{2}\log(\frac{1}{2\pi\sigma^2})-\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}} = -\frac{K}{2}\log(\frac{1}{2\pi\sigma^2})-\sum_{k=1}^{K}\frac{s_k^{2}+m_k^{2}}{2\sigma^{2}}$

which is a function of the variational parameters and hence can be computed.

$\endgroup$
0
$\begingroup$

This might be a bit late but I would like to state what I have understood.

For your first question, the answer is no, the term is $E[log(p(\mu))]$. Here the expectation is over the distribution $q_l(z_l)$. Hence $E_{q(\mu_k)}[log(p(\mu_k))] = \int q(\mu_k; m_k, s_k)log(p(\mu_k)) d\mu_k$ which is not zero(it might be but we have to calculate it). It would have been zero for $E_{p(\mu_k)}[\mu_k]$.

For second term, it should be -log(K) simply because we have assumed the prior $p(c_i)=1/k$ i.e we have considered it to be uniform in the beginning. So our prior tells us that $x_i$ could be from any cluster with same probabilty which is not that useful :) .

About $\phi_{ik}$, I guess they have misprinted it. It says that it uses equation 18, hence the expectation is $E_{-k}$ and not $E$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.