$p(D)$ in Bayesian Statistics

Question

Say I have the following obvious Bayesian computation:

$$ p(\theta|D) = \frac{p(\theta) \cdot p(D|\theta)}{p(D)} $$

where $\theta$ is a model parameter that we try to infer and $D$ is observed data.

I have always understood $p(D)$ to relate to knowledge that we have about the data $D$, but this concept has always been somewhat abstract. Many texts for example, simply ignore $p(D)$.

Moreover, quite often, $p(D)$ is referred to as a normalizing constant. But if this is the case, why write $p(D)$? Is it always the case that $p(D)$ is a constant?

Answer 1

$P(D)$ is not a prior. It is what is called model evidence or marginal likelihood. $P(\theta)$ is the prior over the parameters of interest and $P(D)$ is $\int_{\theta} P(\theta) P(D|\theta) d\theta$. This is basically the normalisation that you need to apply to ensure that the posterior is a valid distribution. So basically we are marginalising out $\theta$ and asking what is the probability of observing $D$.

These are typically difficult to compute. Also see Conjugate priors.

Answer 2

$P(D)$ is necessary if you characterise the full posterior. For example, if you want to do a maximum-a-posteriori (MAP) estimate of your parameters, then you do not need to worry about the normaliser as you are only trying to maximise the posterior probability of the parameters given the observation i.e.

$$ P(\theta|D) \propto P(D|\theta) P(\theta) $$

So, you do not need to worry about the denominator ($P(D)$) as it does not affect finding the $\theta$ that maximizes the posterior. However, MAP gives you a point estimate and you ignore the rich information that posterior distribution may convey.

However, if you want to quantify the uncertainty, do model comparison (see Bayes factor) and probably other things, then you need to also compute or approximate $P(D)$.

I also suggest reading Chris Bishop's book. He explains a lot of these things in an amazing way! The book is called "Pattern Recognition and Machine Learning" by Christopher Bishop. He also has some amazing lectures on probabilistic graphical models and Bayesian inferencing that can be found in the following links:

https://www.youtube.com/watch?v=ju1Grt2hdko

https://www.youtube.com/watch?v=c0AWH5UFyOk

https://www.youtube.com/watch?v=QJSEQeH40hM