5
$\begingroup$

I am reading Bishop's Pattern Recognition and Machine Learning.

In page 73, chapter 2.1. I can't understand the formula 2.19 :

$$p(x=1|\mathcal{D})=\int_0^1 p(x=1|\mu)p(\mu|\mathcal{D})\text{d}\mu $$

The author say, this is obtained by sum and product rules.

The sum rule is:

$$p(X) = \sum_Y p(X,Y)$$

and the product rule is: $$p(X,Y)=p(Y|X)p(X)$$

But from this, I can't deduce the formula. Could you help me please.. thanks very much.

$\endgroup$
6
  • 3
    $\begingroup$ Replace the sum in the sum rule with an integral and then you should be able to derive your result (hint: product rule first, then sum/integral rule). $\endgroup$
    – fabee
    Commented Jul 21, 2014 at 8:45
  • $\begingroup$ @fabee thank you, follow your hint, first I get $p(x=1|\mathcal{D})=\frac{p(\mathcal{D}|x=1)p(x=1)}{p(\mathcal{D})}$ and the integrand $p(x=1|\mu)p(\mu|\mathcal{D})=\frac{p(x=1|\mu)p(\mu|\mathcal{D})p(\mathcal{D})}{p(\mathcal{D})} = \frac{p(x=1|\mu)p(\mu,\mathcal{D})}{p(\mathcal{D})}$, then I need to show that $p(\mathcal{D}|x=1)p(x=1) = \int_0^1p(x=1|\mu)p(\mu,\mathcal{D})\text{d}\mu$. and I can't go any further... $\endgroup$
    – Laura
    Commented Jul 21, 2014 at 14:36
  • 1
    $\begingroup$ Laura, I think this is a good question. You should add the "self-study" tag for questions just like these to get answers just like @fabee's :) $\endgroup$
    – Alexis
    Commented Jul 21, 2014 at 15:52
  • $\begingroup$ @Alexis thank you . I think the author means $p(x=1|\mathcal{D},\mu) = \int_0^1 p(x=1|\mu)p(\mu|\mathcal{D})\text{d}\mu$. but I can't work it out : ) $\endgroup$
    – Laura
    Commented Jul 22, 2014 at 2:56
  • $\begingroup$ @Laura: The equation you wrote in the comment is not correct. First, try to figure out what $p(x=1|\mu)p(\mu|\mathcal D)$ yields (is $\mu$ in front or behind "$|$"?). Hint: This is the application of the product rule. Then try to figure out what happens when you integrate over $\mu$. Can the result still depend on $\mu$ as you wrote? $\endgroup$
    – fabee
    Commented Jul 22, 2014 at 12:40

2 Answers 2

4
$\begingroup$

I just write the answer here, because I have the feeling the comment section is just getting longer without coming to a clear end.

You want to understand the formula

$$p(x=1|\mathcal D) = \int_0^1 p(x=1|\mu)p(\mu|\mathcal D)\mathrm d \mu$$

First, you apply the product rule in the integral. This yields

$$p(x=1|\mathcal D) = \int_0^1 p(x=1,\mu|\mathcal D) \mathrm d \mu$$

This is basically the definition of the summation rule which integrates out $\mu$.

A few comments on that

  1. Note, that in the comments above you said that $\mu$ is in front of "|", but you wrote $p(x=1|\mu,\mathcal D)$. The expression $\int_0^1 p(x=1|\mu, \mathcal D) \mathrm d \mu$ does not give you $p(x=1|\mathcal D)$ since a variable must in front of the conditioning bar "|" before applying the summation rule.
  2. The formula only works if $x$ and $\mathcal D$ are conditionally independent given $\mu$. In the most general case, the equation should be $$p(x=1|\mathcal D) = \int_0^1 p(x=1|\mu,\mathcal D)p(\mu|\mathcal D)\mathrm d \mu.$$
$\endgroup$
2
  • 1
    $\begingroup$ What is "$p(x=1,\mu|\mathcal{D})$" supposed to mean? $\endgroup$
    – whuber
    Commented Jul 22, 2014 at 20:20
  • 1
    $\begingroup$ That's the joint probability of $x$ and $\mu$ given $\mathcal D$ evaluated at $x=1$. I am guessing from Laura's question that $x$ is discrete and $\mu$ is continuous with values in $[0,1]$. $\endgroup$
    – fabee
    Commented Jul 23, 2014 at 9:46
0
$\begingroup$

Just thought I'd add to the above answer by saying that; it makes sense the $x$ and $D$ are conditionally independent given $\mu$. Since if you know the parameter $\mu$, it doesn't matter what your data set is you will know how to calculate $p(x=1)$ using $\mu$ i.e. $p(x=1|\mu, D)=p(x=1|\mu)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.