5
$\begingroup$

From the Wikipedia article on Bayesian inference, we get the following formulation of Bayes' Rule:

$$p(\theta \mid \mathbf{X},\alpha) = \frac{p(\mathbf{X} \mid \theta) p(\theta \mid \alpha)}{p(\mathbf{X} \mid \alpha)} \propto p(\mathbf{X} \mid \theta) p(\theta \mid \alpha).$$

I do not understand this at all. How does one arrive at this equation from the humble $p(A\mid B) = \frac{p(B\mid A)p(A)}{p(B)}$? (I understand Bayes' Rule well in a Frequentist context.)

Also, what is the meaning of the numerator $p(\mathbf{X} \mid \theta) p(\theta \mid \alpha)$, and how is it different from $p(\mathbf{X} \mid \alpha)$?

Improve this question
$\endgroup$
1
  • $\begingroup$ This has nothing to do with the frequentist/Bayesian quarrel. The first formula above is the probabilistic definition of the conditional density of $\theta$ given $X=x$ (I would rather write the above with $x$ instead of $X$) and the hyperparameter $a$. $\endgroup$
    – Xi'an
    Jan 25, 2015 at 17:32

1 Answer 1

Reset to default
5
$\begingroup$

From basic conditional probability ideas:

$\hspace{1cm}p(\theta, X,\alpha) = p(\theta\mid X,\alpha)\, p(X,\alpha)$

also

$\hspace{1cm}p(\theta, X,\alpha) = p(X \mid \theta,\alpha)\, p(\theta,\alpha)$

Hence

$\hspace{1cm}p(\theta\mid X,\alpha) = \frac{p(X \mid \theta,\alpha)\, p(\theta,\alpha)}{p(X,\alpha)}$

$\hspace{3.3cm} = \frac{p(X \mid \theta,\alpha)\, p(\theta|\alpha)\, p(\alpha)}{p(X,\alpha)}$

$\hspace{3.3cm}\propto p(X \mid \theta,\alpha)\, p(\theta|\alpha)\, p(\alpha)$

But here $\alpha$ appears to be fixed/known, and given $\theta$, $X$ doesn't depend on $\alpha$ (i.e. $X$ is only related to $\alpha$ via $\theta$ ... $p(X|\theta,\alpha)=p(X|\theta)$), so the result then follows:

$\hspace{3.3cm}\propto p(X \mid \theta)\, p(\theta|\alpha)$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your final expression is $p(x\mid \theta,a)p(\theta\mid a)p(a)$. How does this relate to $p(x\mid \theta)p(\theta\mid a)$? $\endgroup$
    – Simon Kuang
    Nov 19, 2014 at 5:28
  • $\begingroup$ Okay, small edit made. $\endgroup$
    – Glen_b
    Nov 19, 2014 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.