This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$ L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ $$ $$ =\int q_{j}\left[\int \ln p(X,Z) \prod_{i\neq j}q_{i} dZ_{i}\right] dZ_{j} - \int q_{j}\ln(q_{j})dZ_{j} + const $$ $$ =\int q_j \ln\tilde{p}(X,Z_{j})dZ_{j} - \int q_{j} \ln(q_{j}) dZ_{j} + const $$

where $\ln(\tilde{p}(X,Z_{j}) = \mathbb{E}_{i \neq j}[\ln(p(X,Z)] + const$ where the expectation is with respect to the q distributions over all variables $z_{i}$ for $i \neq j$

Since $\prod_{i\neq j}q_{i}$ multiplies both the terms within { } in the first part of equation 10.6 shouldn't the last equation of 10.6 read as follows?

$$\int q_j\ln\tilde{p}(X,Z_{j})dZ_{j} - (\prod_{i\neq j}q_{j}) \int q_{j} \ln(q_{j}) dZ_{j} + const$$

This would then no longer be the negative KL divergence between $q_{j}(Z_{j})$ and $\tilde{p}(X,Z_{j})$ as is claimed later in the text. What am i missing?

  • $\begingroup$ Welcome to the community. Could you please include the equation as it is stated in the Bishop book in your question? $\endgroup$ – Ruben van Bergen Apr 3 at 14:35
  • 1
    $\begingroup$ Hi Ruben van Bergen, thanks for the response, have edited my question with the equation 10.6 at the beginning. $\endgroup$ – STEMExchanger Apr 3 at 15:03

As you might notice, in the last equation your integration variable is not $dZ$ but $dZ_j$. It should already give you an idea how the last equation appears.

The variables $q_j$ represent distributions. In fact when you integrate the equation $\int\prod_iq_i\ln(q_j)dZ$, it is the same as integrating

$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j}$

You are simply integrating the terms $\prod_{i \neq j}q_i$ out by marginalizing, since each $q_i$ is a distribution, which becomes one when integrated. Thus, you have

$\quad\quad\int q_j\ln(q_j)\prod_{i \neq j}q_i dZ_jdZ_{i \neq j} = \int q_j\ln(q_j)dZ_j\int \prod_{i \neq j}q_i dZ_{i \neq j} = \int q_j\ln(q_j)dZ_j $

The term $\int \prod_{i \neq j}q_i dZ_{i \neq j}$ becomes one.

  • $\begingroup$ Got it, in your expression after "integrate the equation" shouldn't the integral come up front instead of after $q_{i}$, also i presume your last expression in your last statement in the answer is missing an integral $\endgroup$ – STEMExchanger Apr 3 at 15:57
  • $\begingroup$ @STEMExchanger, yep, glad I could help :) $\endgroup$ – SWIM S. Apr 3 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.