# Why is a normalizing factor required in Bayes’ Theorem?

Bayes theorem goes $$P(\textrm{model}|\textrm{data}) = \frac{P(\textrm{model}) \times P(\textrm{data}|\textrm{model})}{P(\textrm{data})}$$

This is all fine. But, I've read somewhere:

Basically, P(data) is nothing but a normalising constant, i.e., a constant that makes the posterior density integrate to one.

We know that $0 \leq P(\textrm{model}) \leq 1$ and $0 \leq P(\textrm{data}|\textrm{model}) \leq 1$.

Therefore, $P(\textrm{model}) \times P(\textrm{data}|\textrm{model})$ must be between 0 and 1 as well. In such a case, why do we need a normalizing constant to make the posterior integrate to one?

• When you are working with probability densities, as mentioned in this post, you can no longer conclude 0 <= P(model) <= 1 nor 0 <= P(data/model) <= 1, because either (or even both!) of those could exceed $1$ (and even be infinite). See stats.stackexchange.com/questions/4220.
– whuber
Dec 18, 2014 at 22:35
• It is not the case that $$P(\textrm{data}|\textrm{model})\le 1$$ because this vague notation represents the integrated likelihood of the data, not a probability. Dec 19, 2014 at 11:29

First, the integral of "likelihood x prior" is not necessarily 1.

It is not true that if:

$$0 \leq P(\textrm{model}) \leq 1$$ and $$0 \leq P(\textrm{data}|\textrm{model}) \leq 1$$

then the integral of this product with respect to the model (to the parameters of the model, indeed) is 1.

Demonstration. Imagine two discrete densities: $$P(\textrm{model}) = [0.5, 0.5] \text{ (this is called "prior")}\\ P(\textrm{data | model}) = [0.80, 0.2] \text{ (this is called "likelihood")}\\$$

If you multiply them both you get: $$[0.40, 0.25]$$ which is not a valid density since it does not integrate to one: $$0.40 + 0.25 = 0.65$$

So, what should we do to force the integral to be 1? Use the normalizing factor, which is: $$\sum_{\text{model_params}} P(\text{model}) P(\text{data | model}) = \sum_\text{model_params} P(\text{model, data}) = P(\text{data}) = 0.65$$

(sorry about the poor notation. I wrote three different expressions for the same thing since you might see them all in the literature)

Second, the "likelihood" can be anything, and even if it is a density, it can have values higher than 1.

As @whuber said this factors do not need to be between 0 and 1. They need that their integral (or sum) be 1.

You will often see: $$P(\textrm{model}|\textrm{data}) \propto P(\textrm{data}|\textrm{model}) P(\text{model})$$ because the missing denominator can be easily get by integrating this product. Note that this integration will have one well known result if the prior and the likelihood are conjugate.

• +1. This is the only answer that actually addresses the original question of why the normalization constant is needed to make the posterior integrate to one. What you do with the posterior later (e.g. MCMC inference or calculating absolute probabilities) is a different matter. May 30, 2017 at 13:07
• I have difficulties following the notation. What does it mean $P(model)=[0.5,0.5]$? Suppose we use a Gaussian model with given variance $\sigma^2=1$, and we wish to find the mean $\mu$. What does it mean $P(\mu) = [0.5, 0.5]$? Apr 27, 2019 at 9:57
• I tried to follow the notation of the OP. I mean that you have two possible values for your model parameter, and that both are equally likely. For instance, is there a teapot revolving around the Earth? yes/no? In your case it does not make sense because your parameter $\mu$ is continuous and can take infinite values. May 4, 2019 at 19:01
• Sorry to ask a silly question. $$P(\textrm{model}) = [0.5, 0.5] \text{ (this is called "prior")}\\ P(\textrm{data | model}) = [0.80, 0.2] \text{ (this is called "likelihood")}\\$$ If you multiply them both you get: $$[0.40, 0.25]$$ How did you multiply? I tried to multiply lower limit with lower $(0.5\times 0.8)$, and upper limit with upper $(0.5\times 0.2)$, which doesn't yield $[0.40, 0.25]$. Aug 15 at 14:27
• @user31466 you are right, it should be [0.4, 0.1]. Aug 24 at 18:58

The short answer to your question is that without the denominator, the expression on the right-hand side is merely a likelihood, not a probability, which can only range from 0 to 1. The "normalizing constant" allows us to get the probability for the occurrence of an event, rather than merely the relative likelihood of that event compared to another.

Bayes theorem is often defined as:

$$P(\text{model}~ | ~\text{data}) \propto P(\text{model}) \times P(\text{data}~|~\text{model})$$

because the only reason why you need the constant is so that it integrates to 1 (see the answers by others). This is not needed in most MCMC simulation approaches to Bayesian analysis and hence the constant is dropped from the equation. So for most simulations it is not even required.

I love the description by Kruschke: the last puppy (constant) is sleepy because he has nothing to do in the formula. Also some, like Andrew Gelman, consider the constant as "overrated" and "basically meaningless when people use flat priors" (check the discussion here).

• +1 to the introduction of puppies. "No animals were harmed in the writing of this answer" :) Dec 19, 2014 at 8:50

To help you understand why $$p(\text{data})$$ is a normalization constant, I will slightly change the notation to make it clear that it holds for any two random variables $$X$$ and $$Y$$ (whose realizations I will denote by $$x$$ and $$y$$ respectively).

With this, we have:

$$p(x|y) = \frac{p(x) \cdot p(y | x)}{p(y)}$$

Now, the following equality holds for $$p(y)$$:

$$p(y) = \int p(x, y) dx = \int p(x) \cdot p(y|x)dx$$

Plugging this equation into the first equation yields:

$$p(x|y) = \frac{p(x)\cdot p(y|x)}{\int p(x) \cdot p(y|x)dx}$$

Now it's clear why $$p(y)$$ $$-$$ or with your notation $$p(\text{data})$$ $$-$$ is a normalization constant.