3
$\begingroup$

I'm reading a book about Bayesian statistics and at some stage it explain the Bayes' rule as follow: $$p(\theta|D) = \frac{p(D|\theta)\,p(\theta)}{p(D)}$$

Where $\theta$ is the model parameter and D is the data. Until here I can understand.

Then it says "We talk about parameter values $\theta$ only in the context of a particular model that gives the meaning to the parameter. In some application it can help to make the model explicit in Bayes' rule. Let's call the model $M$. Then, because all probabilities are defined given that model, we can rewrite Eq 4.4 as:"

$$p(\theta|D,M) = \frac{p(D|\theta, M)\,p(\theta|M)}{p(D|M)}$$

Until now my understanding is that $P(A,B)$ is the co-joint probability of two events happening at the same time, so I intuitively interpret $p(\theta|D,M)$ as the probability of $\theta$ given $D$ in the context of the model $M$. The same applies for $p(D|\theta,M)$.

I'm lost when it multiplies that value by $p(\theta|M)$, where does it come from? I would expect that to be $p(\theta,M)$ (i.e.: $\theta$ in the context of model $M$).

It is very likely that my intuitively interpretation is wrong, could you please help me understand how the first equation connects to the second?

$\endgroup$
2
  • $\begingroup$ Could you provide the reference of the book you quote from? $\endgroup$
    – Xi'an
    Jul 19, 2016 at 16:33
  • $\begingroup$ Yes, it's "Doing Bayesian Data Analysis" by John Kruschke $\endgroup$ Jul 19, 2016 at 16:40

1 Answer 1

3
$\begingroup$

The two equations are equivalent. The second equation just shows the dependence on the model explicitly. In the first equation, all values do depend on the model, but it's implicit.

Letting dependence on the model be implicit for the moment, we can derive Bayes' rule as follows. By the definition of conditional probability:

$$p(\theta \mid D) = \frac{p(\theta, D)}{p(D)}$$

So we can write the joint distribution as:

$$p(\theta, D) = p(\theta \mid D) p(D)$$

By similar logic, we can also write the joint distribution as:

$$p(\theta, D) = p(D \mid \theta) p(\theta)$$

Setting these two expressions equal, we have:

$$p(\theta \mid D) p(D) = p(D \mid \theta) p(\theta)$$

Divide both sides by $p(D)$ to obtain Bayes' rule:

$$p(\theta \mid D) = \frac{p(D \mid \theta) p(\theta)}{p(D)}$$

Now, let's make dependence on the model explicit. Write the joint distribution of $\theta$ and $D$ (given the model) as:

$p(\theta, D \mid M) = p(\theta \mid D, M) p(D \mid M) = p(D \mid \theta, M) p(\theta \mid M)$

As before, this follows from the definition of conditional probability. Divide by $p(D \mid M)$ to obtain Bayes' rule:

$$p(\theta \mid D, M) = \frac{p(D \mid \theta, M) p(\theta \mid M)}{p(D \mid M)}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.