Help with derivation of Mean Field Variational Inference

Question

I am studying Variational Inference using Bishop's book: Pattern Recognition and Machine Learning. At the moment, I am struggling to understand the Lower Bound derivation for the Mean-Field Variational inference at page 465, equation 10.6.

Surely I am missing some basic algebraic detail.. but, I can't for the moment see how this $\int \prod_{i}q_{i}\{\ln{p(\boldsymbol{X},\boldsymbol{Z})}\}d\boldsymbol{Z}$ turns into $\int q_{j}\{ \int \ln{p(\boldsymbol{X},\boldsymbol{Z})} \prod_{i \neq j} q_{i}\,d \boldsymbol{Z_{i}}\} \, d\boldsymbol{Z_{j}}$. From where does the extra integral come from? I have some ideas, but I am not sure of those. So, I want to read some intuitive explanation (with math as well) on the subject.

Additionally, can someone gently expose the complete algebraic derivation of equation 10.6 including showing the constant values?

Answer 1

The main issue is that the integrals involved are multivariate. A confusing thing about Bishop's notation is that, inside those integrals, $q_i$ should actually be $q_i(\mathbf{Z}_i)$.

So we want to maximize the bound $$\mathcal{L}(q) = \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})]-\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] $$ with respect to the $j$-th marginal of the distribution $q$.

Because of the mean-field assumption, $$\log q(\mathbf{Z})= \log q_j(\mathbf{Z}_j) + \sum_{i \neq j} \log q_i(\mathbf{Z}_i),$$ therefore the entropy term will be: $$\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] = \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j} \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)].$$

Now, for the first term of the sum, there is actually no extra integral, it's just that we're considering a multivariate integral. We'll use the mean field assumption to break the multivariate integral: $$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z}) q(\mathbf{Z})d\mathbf{Z}\\=\int \log p(\mathbf{X},\mathbf{Z}) \prod_i q_i(\mathbf{Z}_i)d\mathbf{Z}_i \\= \int \left( \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\= \int \left( \mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})]\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j,$$ using the notation that Bishop introduces in Formula (10.8): $$\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i.$$ Now, denoting $$A = \int \exp(\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})])d\mathbf{Z}_j,$$ we can write: $$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] =\int \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})A/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\= \int \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j +\log(A).$$ Note that, in Bishop's notations, we have exactly: $$\tilde{p}(\mathbf{X},\mathbf{Z}_j) = \exp \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)].$$ By combining everything, we end up with: $$\mathcal{L}(q) = \int \log \tilde{p}(\mathbf{X},\mathbf{Z}_j)q_j(\mathbf{Z}_j)d\mathbf{Z}_j + \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j} \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)] + \log(A).$$ where the two last terms are the "constants" that do not depend on $q_j$.