I am studying Variational Inference using Bishop's book: Pattern Recognition and Machine Learning. At the moment, I am struggling to understand the Lower Bound derivation for the Mean-Field Variational inference at page 465, equation 10.6.

Surely I am missing some basic algebraic detail.. but, I can't for the moment see how this $\int \prod_{i}q_{i}\{\ln{p(\boldsymbol{X},\boldsymbol{Z})}\}d\boldsymbol{Z}$ turns into $\int q_{j}\{ \int \ln{p(\boldsymbol{X},\boldsymbol{Z})} \prod_{i \neq j} q_{i}\,d \boldsymbol{Z_{i}}\} \, d\boldsymbol{Z_{j}}$. From where does the extra integral come from? I have some ideas, but I am not sure of those. So, I want to read some intuitive explanation (with math as well) on the subject.

Additionally, can someone gently expose the complete algebraic derivation of equation 10.6 including showing the constant values?

  • $\begingroup$ "Additionally, can someone gently expose the complete algebraic derivation of equation 10.6 including showing the constant values?" Leaving that formula out of your question limits the list of someones to owners of the book. Could you add it? $\endgroup$ – Sean Easter Dec 8 '17 at 17:25
  • 1
    $\begingroup$ The book has been released microsoft.com/en-us/research/people/cmbishop/#!prml-book $\endgroup$ – adn Nov 29 '18 at 18:42

The main issue is that the integrals involved are multivariate. A confusing thing about Bishop's notation is that, inside those integrals, $q_i$ should actually be $q_i(\mathbf{Z}_i)$.

So we want to maximize the bound $$\mathcal{L}(q) = \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})]-\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] $$ with respect to the $j$-th marginal of the distribution $q$.

Because of the mean-field assumption, $$\log q(\mathbf{Z})= \log q_j(\mathbf{Z}_j) + \sum_{i \neq j} \log q_i(\mathbf{Z}_i),$$ therefore the entropy term will be: $$\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] = \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j} \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)].$$

Now, for the first term of the sum, there is actually no extra integral, it's just that we're considering a multivariate integral. We'll use the mean field assumption to break the multivariate integral: $$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z}) q(\mathbf{Z})d\mathbf{Z}\\=\int \log p(\mathbf{X},\mathbf{Z}) \prod_i q_i(\mathbf{Z}_i)d\mathbf{Z}_i \\= \int \left( \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\= \int \left( \mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})]\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j,$$ using the notation that Bishop introduces in Formula (10.8): $$\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i.$$ Now, denoting $$A = \int \exp(\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})])d\mathbf{Z}_j,$$ we can write: $$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] =\int \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})A/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\= \int \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j +\log(A).$$ Note that, in Bishop's notations, we have exactly: $$\tilde{p}(\mathbf{X},\mathbf{Z}_j) = \exp \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)].$$ By combining everything, we end up with: $$\mathcal{L}(q) = \int \log \tilde{p}(\mathbf{X},\mathbf{Z}_j)q_j(\mathbf{Z}_j)d\mathbf{Z}_j + \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j} \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)] + \log(A).$$ where the two last terms are the "constants" that do not depend on $q_j$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.