# Equation 10.6 in Bishop book

This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ$$ $$=\int q_{j}\left[\int \ln p(X,Z) \prod_{i\neq j}q_{i} dZ_{i}\right] dZ_{j} - \int q_{j}\ln(q_{j})dZ_{j} + const$$ $$=\int q_j \ln\tilde{p}(X,Z_{j})dZ_{j} - \int q_{j} \ln(q_{j}) dZ_{j} + const$$

where $$\ln(\tilde{p}(X,Z_{j}) = \mathbb{E}_{i \neq j}[\ln(p(X,Z)] + const$$ where the expectation is with respect to the q distributions over all variables $$z_{i}$$ for $$i \neq j$$

Since $$\prod_{i\neq j}q_{i}$$ multiplies both the terms within { } in the first part of equation 10.6 shouldn't the last equation of 10.6 read as follows?

$$\int q_j\ln\tilde{p}(X,Z_{j})dZ_{j} - (\prod_{i\neq j}q_{j}) \int q_{j} \ln(q_{j}) dZ_{j} + const$$

This would then no longer be the negative KL divergence between $$q_{j}(Z_{j})$$ and $$\tilde{p}(X,Z_{j})$$ as is claimed later in the text. What am i missing?

• Welcome to the community. Could you please include the equation as it is stated in the Bishop book in your question? – Ruben van Bergen Apr 3 at 14:35
• Hi Ruben van Bergen, thanks for the response, have edited my question with the equation 10.6 at the beginning. – STEMExchanger Apr 3 at 15:03

As you might notice, in the last equation your integration variable is not $$dZ$$ but $$dZ_j$$. It should already give you an idea how the last equation appears.

The variables $$q_j$$ represent distributions. In fact when you integrate the equation $$\int\prod_iq_i\ln(q_j)dZ$$, it is the same as integrating

$$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j}$$

You are simply integrating the terms $$\prod_{i \neq j}q_i$$ out by marginalizing, since each $$q_i$$ is a distribution, which becomes one when integrated. Thus, you have

$$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j} = \int q_j\ln(q_j)dZ_j\int \prod_{i \neq j}q_i dZ_{i \neq j} = \int q_j\ln(q_j)dZ_j$$

The term $$\int \prod_{i \neq j}q_i dZ_{i \neq j}$$ becomes one.

• Got it, in your expression after "integrate the equation" shouldn't the integral come up front instead of after $q_{i}$, also i presume your last expression in your last statement in the answer is missing an integral – STEMExchanger Apr 3 at 15:57
• @STEMExchanger, yep, glad I could help :) – SWIM S. Apr 3 at 16:17