# sum and product rules of probability

I am reading Bishop's Pattern Recognition and Machine Learning.

In page 73, chapter 2.1. I can't understand the formula 2.19 :

$$p(x=1|\mathcal{D})=\int_0^1 p(x=1|\mu)p(\mu|\mathcal{D})\text{d}\mu$$

The author say, this is obtained by sum and product rules.

The sum rule is:

$$p(X) = \sum_Y p(X,Y)$$

and the product rule is: $$p(X,Y)=p(Y|X)p(X)$$

But from this, I can't deduce the formula. Could you help me please.. thanks very much.

• Replace the sum in the sum rule with an integral and then you should be able to derive your result (hint: product rule first, then sum/integral rule). Commented Jul 21, 2014 at 8:45
• @fabee thank you, follow your hint, first I get $p(x=1|\mathcal{D})=\frac{p(\mathcal{D}|x=1)p(x=1)}{p(\mathcal{D})}$ and the integrand $p(x=1|\mu)p(\mu|\mathcal{D})=\frac{p(x=1|\mu)p(\mu|\mathcal{D})p(\mathcal{D})}{p(\mathcal{D})} = \frac{p(x=1|\mu)p(\mu,\mathcal{D})}{p(\mathcal{D})}$, then I need to show that $p(\mathcal{D}|x=1)p(x=1) = \int_0^1p(x=1|\mu)p(\mu,\mathcal{D})\text{d}\mu$. and I can't go any further... Commented Jul 21, 2014 at 14:36
• Laura, I think this is a good question. You should add the "self-study" tag for questions just like these to get answers just like @fabee's :) Commented Jul 21, 2014 at 15:52
• @Alexis thank you . I think the author means $p(x=1|\mathcal{D},\mu) = \int_0^1 p(x=1|\mu)p(\mu|\mathcal{D})\text{d}\mu$. but I can't work it out : ) Commented Jul 22, 2014 at 2:56
• @Laura: The equation you wrote in the comment is not correct. First, try to figure out what $p(x=1|\mu)p(\mu|\mathcal D)$ yields (is $\mu$ in front or behind "$|$"?). Hint: This is the application of the product rule. Then try to figure out what happens when you integrate over $\mu$. Can the result still depend on $\mu$ as you wrote? Commented Jul 22, 2014 at 12:40

I just write the answer here, because I have the feeling the comment section is just getting longer without coming to a clear end.

You want to understand the formula

$$p(x=1|\mathcal D) = \int_0^1 p(x=1|\mu)p(\mu|\mathcal D)\mathrm d \mu$$

First, you apply the product rule in the integral. This yields

$$p(x=1|\mathcal D) = \int_0^1 p(x=1,\mu|\mathcal D) \mathrm d \mu$$

This is basically the definition of the summation rule which integrates out $\mu$.

1. Note, that in the comments above you said that $\mu$ is in front of "|", but you wrote $p(x=1|\mu,\mathcal D)$. The expression $\int_0^1 p(x=1|\mu, \mathcal D) \mathrm d \mu$ does not give you $p(x=1|\mathcal D)$ since a variable must in front of the conditioning bar "|" before applying the summation rule.
2. The formula only works if $x$ and $\mathcal D$ are conditionally independent given $\mu$. In the most general case, the equation should be $$p(x=1|\mathcal D) = \int_0^1 p(x=1|\mu,\mathcal D)p(\mu|\mathcal D)\mathrm d \mu.$$
• What is "$p(x=1,\mu|\mathcal{D})$" supposed to mean?
• That's the joint probability of $x$ and $\mu$ given $\mathcal D$ evaluated at $x=1$. I am guessing from Laura's question that $x$ is discrete and $\mu$ is continuous with values in $[0,1]$. Commented Jul 23, 2014 at 9:46
Just thought I'd add to the above answer by saying that; it makes sense the $$x$$ and $$D$$ are conditionally independent given $$\mu$$. Since if you know the parameter $$\mu$$, it doesn't matter what your data set is you will know how to calculate $$p(x=1)$$ using $$\mu$$ i.e. $$p(x=1|\mu, D)=p(x=1|\mu)$$.