Why is a normalizing factor required in Bayes’ Theorem?

Question

Bayes theorem goes $$ P(\textrm{model}|\textrm{data}) = \frac{P(\textrm{model}) \times P(\textrm{data}|\textrm{model})}{P(\textrm{data})} $$

This is all fine. But, I've read somewhere:

Basically, P(data) is nothing but a normalising constant, i.e., a constant that makes the posterior density integrate to one.

We know that $0 \leq P(\textrm{model}) \leq 1$ and $ 0 \leq P(\textrm{data}|\textrm{model}) \leq 1$.

Therefore, $P(\textrm{model}) \times P(\textrm{data}|\textrm{model})$ must be between 0 and 1 as well. In such a case, why do we need a normalizing constant to make the posterior integrate to one?

Answer 1

First, the integral of "likelihood x prior" is not necessarily 1.

It is not true that if:

$0 \leq P(\textrm{model}) \leq 1$ and $ 0 \leq P(\textrm{data}|\textrm{model}) \leq 1$

then the integral of this product with respect to the model (to the parameters of the model, indeed) is 1.

Demonstration. Imagine two discrete densities: $$ P(\textrm{model}) = [0.5, 0.5] \text{ (this is called "prior")}\\ P(\textrm{data | model}) = [0.80, 0.2] \text{ (this is called "likelihood")}\\ $$

If you multiply them both you get: $$ [0.40, 0.25] $$ which is not a valid density since it does not integrate to one: $$ 0.40 + 0.25 = 0.65 $$

So, what should we do to force the integral to be 1? Use the normalizing factor, which is: $$ \sum_{\text{model_params}} P(\text{model}) P(\text{data | model}) = \sum_\text{model_params} P(\text{model, data}) = P(\text{data}) = 0.65 $$

(sorry about the poor notation. I wrote three different expressions for the same thing since you might see them all in the literature)

Second, the "likelihood" can be anything, and even if it is a density, it can have values higher than 1.

As @whuber said this factors do not need to be between 0 and 1. They need that their integral (or sum) be 1.

Third [extra], "conjugates" are your friends to help you find the normalizing constant.

You will often see: $$ P(\textrm{model}|\textrm{data}) \propto P(\textrm{data}|\textrm{model}) P(\text{model}) $$ because the missing denominator can be easily get by integrating this product. Note that this integration will have one well known result if the prior and the likelihood are conjugate.

Answer 2

The short answer to your question is that without the denominator, the expression on the right-hand side is merely a likelihood, not a probability, which can only range from 0 to 1. The "normalizing constant" allows us to get the probability for the occurrence of an event, rather than merely the relative likelihood of that event compared to another.

Answer 3

You already got two valid answers but let me add my two cents.

Bayes theorem is often defined as:

$$P(\text{model}~ | ~\text{data}) \propto P(\text{model}) \times P(\text{data}~|~\text{model})$$

because the only reason why you need the constant is so that it integrates to 1 (see the answers by others). This is not needed in most MCMC simulation approaches to Bayesian analysis and hence the constant is dropped from the equation. So for most simulations it is not even required.

I love the description by Kruschke: the last puppy (constant) is sleepy because he has nothing to do in the formula.

Also some, like Andrew Gelman, consider the constant as "overrated" and "basically meaningless when people use flat priors" (check the discussion here).

Answer 4

To help you understand why $p(\text{data})$ is a normalization constant, I will slightly change the notation to make it clear that it holds for any two random variables $X$ and $Y$ (whose realizations I will denote by $x$ and $y$ respectively).

With this, we have:

$$p(x|y) = \frac{p(x) \cdot p(y | x)}{p(y)}$$

Now, the following equality holds for $p(y)$:

$$p(y) = \int p(x, y) dx = \int p(x) \cdot p(y|x)dx$$

Plugging this equation into the first equation yields:

$$p(x|y) = \frac{p(x)\cdot p(y|x)}{\int p(x) \cdot p(y|x)dx}$$

Now it's clear why $p(y)$ $-$ or with your notation $p(\text{data})$ $-$ is a normalization constant.