Help with derivation of Mean Field Variational Inference I am studying Variational Inference using Bishop's book: Pattern Recognition and Machine Learning. At the moment, I am struggling to understand the Lower Bound derivation for the Mean-Field Variational inference at page 465, equation 10.6.
Surely I am missing some basic algebraic detail.. but, I can't for the moment see how this $\int \prod_{i}q_{i}\{\ln{p(\boldsymbol{X},\boldsymbol{Z})}\}d\boldsymbol{Z}$ turns into $\int q_{j}\{ \int \ln{p(\boldsymbol{X},\boldsymbol{Z})} \prod_{i \neq j} q_{i}\,d \boldsymbol{Z_{i}}\} \, d\boldsymbol{Z_{j}}$. From where does the extra integral come from? I have some ideas, but I am not sure of those. So, I want to read some intuitive explanation (with math as well) on the subject.
Additionally, can someone gently expose the complete algebraic derivation of equation 10.6 including showing the constant values?   
 A: The main issue is that the integrals involved are multivariate. A confusing thing about Bishop's notation is that, inside those integrals, $q_i$ should actually be $q_i(\mathbf{Z}_i)$.
So we want to maximize the bound
$$\mathcal{L}(q) = \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})]-\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] $$
with respect to the $j$-th marginal of the distribution $q$.
Because of the mean-field assumption, $$\log q(\mathbf{Z})=  \log q_j(\mathbf{Z}_j) + \sum_{i \neq j}  \log q_i(\mathbf{Z}_i),$$
therefore the entropy term will be:
$$\mathbb{E}_{\mathbf{Z}\sim q}[\log q(\mathbf{Z})] =  \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j}  \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)].$$
Now, for the first term of the sum, there is actually no extra integral, it's just that we're considering a multivariate integral. We'll use the mean field assumption to break the multivariate integral:
$$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z}) q(\mathbf{Z})d\mathbf{Z}\\=\int \log p(\mathbf{X},\mathbf{Z}) \prod_i q_i(\mathbf{Z}_i)d\mathbf{Z}_i \\= \int  \left( \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\=
\int \left( \mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})]\right) q_j(\mathbf{Z}_j)d\mathbf{Z}_j,$$
using the notation that Bishop introduces in Formula (10.8):
$$\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})] = \int \log p(\mathbf{X},\mathbf{Z})\prod_{i\neq j}q_i(\mathbf{Z}_i)d\mathbf{Z}_i.$$
Now, denoting
$$A = \int \exp(\mathbb{E}_{i \neq j}[\log p(\mathbf{X},\mathbf{Z})])d\mathbf{Z}_j,$$
we can write:
$$ \mathbb{E}_{\mathbf{Z}\sim q}[\log p(\mathbf{X},\mathbf{Z})] =\int  \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})A/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j \\= \int  \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)]q_j(\mathbf{Z}_j)d\mathbf{Z}_j +\log(A).$$
Note that, in Bishop's notations, we have exactly:
$$\tilde{p}(\mathbf{X},\mathbf{Z}_j) = \exp \mathbb{E}_{i \neq j}[\log (p(\mathbf{X},\mathbf{Z})/A)].$$
By combining everything, we end up with: 
$$\mathcal{L}(q) = \int  \log \tilde{p}(\mathbf{X},\mathbf{Z}_j)q_j(\mathbf{Z}_j)d\mathbf{Z}_j + \mathbb{E}_{\mathbf{Z}_j\sim q_j}[\log q(\mathbf{Z}_j)]+ \sum_{i \neq j}  \mathbb{E}_{\mathbf{Z}_i\sim q_i}[\log q(\mathbf{Z}_i)] + \log(A).$$
where the two last terms are the "constants" that do not depend on $q_j$.
