Equation 10.6 in Bishop book

Question

This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$ L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ $$ $$ =\int q_{j}\left[\int \ln p(X,Z) \prod_{i\neq j}q_{i} dZ_{i}\right] dZ_{j} - \int q_{j}\ln(q_{j})dZ_{j} + const $$ $$ =\int q_j \ln\tilde{p}(X,Z_{j})dZ_{j} - \int q_{j} \ln(q_{j}) dZ_{j} + const $$

where $\ln(\tilde{p}(X,Z_{j}) = \mathbb{E}_{i \neq j}[\ln(p(X,Z)] + const$ where the expectation is with respect to the q distributions over all variables $z_{i}$ for $i \neq j$

Since $\prod_{i\neq j}q_{i}$ multiplies both the terms within { } in the first part of equation 10.6 shouldn't the last equation of 10.6 read as follows?

$$\int q_j\ln\tilde{p}(X,Z_{j})dZ_{j} - (\prod_{i\neq j}q_{j}) \int q_{j} \ln(q_{j}) dZ_{j} + const$$

This would then no longer be the negative KL divergence between $q_{j}(Z_{j})$ and $\tilde{p}(X,Z_{j})$ as is claimed later in the text. What am i missing?

Answer 1

As you might notice, in the last equation your integration variable is not $dZ$ but $dZ_j$. It should already give you an idea how the last equation appears.

The variables $q_j$ represent distributions. In fact when you integrate the equation $\int\prod_iq_i\ln(q_j)dZ$, it is the same as integrating

$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j}$

You are simply integrating the terms $\prod_{i \neq j}q_i$ out by marginalizing, since each $q_i$ is a distribution, which becomes one when integrated. Thus, you have

$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j} = \int q_j\ln(q_j)dZ_j\int \prod_{i \neq j}q_i dZ_{i \neq j} = \int q_j\ln(q_j)dZ_j $

The term $\int \prod_{i \neq j}q_i dZ_{i \neq j}$ becomes one.