I'm currently reading Bishops: Pattern Matching and Machine Learning and ran into an equation I didn't grasp. Page 373, Graphical Models, section 8.2.1.

He says:


Given the joint distribution:

\begin{equation} p(a,b,c) = p(a|c)p(b|c)p(c) \end{equation} We can investigate whether a and b are independent by marginalizing both sides with respect to c, \begin{equation} p(a,b) = \sum_c p(a|c)p(b|c)p(c) \end{equation} and in general this does NOT factorize to the product: \begin{equation} p(a)p(b) \end{equation}


My question is why it would not factor into the product above? Seems reasonable that once c has been marginalized out that's exactly the product you would get?

Summing over all values of c should result in: \begin{equation} \sum_c p(c) = 1 \end{equation} and thus we should arrive at the product, \begin{equation} p(a,b) = \sum_c p(a|c)p(b|c)=p(a)p(b) \end{equation} What am I missing?


Note that, in general, $$\sum_c p(a|c) p(b|c) p(c) \ne p(a) p(b) \sum_c p(c)\,. $$

The fact that $\sum_c p(c) = 1$ can only be used if $p(a|c) = p(a)$ and $p(b|c) = p(b)$, which is not always true. In general $\sum_c p(c) = 1$ will not be useful. Consider for example, consider for $c \in \{1,2\}$

$$p(a|c) = ce^{-ca}, p(b|c) = ce^{-cb}, \text{ and } p(c) = 1/2$$

Then \begin{align*} p(a,b) & = \sum_c p(a|c) p(b|c) p(c)\\ & = \sum_c \dfrac{ce^{-ca} ce^{-cb}}{2} \\ & = \sum_c \dfrac{c^2 e^{-c(a+b)}}{2}\\ & = \dfrac{e^{-(a+b)}}{2} + \dfrac{4e^{-2(a+b)}}{2}\\ & = \dfrac{e^{-(a+b)}}{2}\left(1 + 4e^{-(a+b)} \right) \end{align*}

The above cannot be factored in $p(a)p(b)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.