23
$\begingroup$

I read that in Bayes rule, the denominator $\Pr(\textrm{data})$ of

$$\Pr(\text{parameters} \mid \text{data}) = \frac{\Pr(\textrm{data} \mid \textrm{parameters}) \Pr(\text{parameters})}{\Pr(\text{data})}$$

is called a normalizing constant. What exactly is it? What is its purpose? Why does it look like $\Pr(data)$? Why doesn't it depend on the parameters?

Improve this question
$\endgroup$
1
  • 6
    $\begingroup$ When you integrate $f(\text{data}|\text{params})f(\text{params})$, you are integrating over the parameters and so the result has no term depending on the parameters, in the same way that $\int_{x=0}^{x=2}xy\;dx = 2y$ does not depend on $x$. $\endgroup$
    – Henry
    Commented Jun 20, 2011 at 18:57

3 Answers 3

Reset to default
22
$\begingroup$

The denominator, $\Pr(\textrm{data})$, is obtained by integrating out the parameters from the join probability, $\Pr(\textrm{data}, \textrm{parameters})$. This is the marginal probability of the data and, of course, it does not depend on the parameters since these have been integrated out.

Now, since:

  • $\Pr(\textrm{data})$ does not depend on the parameters for which one wants to make inference;
  • $\Pr(\textrm{data})$ is generally difficult to calculate in a closed-form;

one often uses the following adaptation of Baye's formula:

$\Pr(\textrm{parameters} \mid \textrm{data}) \propto \Pr(\textrm{data} \mid \textrm{parameters}) \Pr(\textrm{parameters})$

Basically, $\Pr(\textrm{data})$ is nothing but a "normalising constant", i.e., a constant that makes the posterior density integrate to one.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ @nbro: I mean Pr(data) = integral over the parameters of Pr(data, parameters) $\endgroup$
    – ocram
    Commented Mar 8, 2018 at 6:24
  • $\begingroup$ What do you mean by 'P(data) is generally difficult to calculate in a closed-form'? $\endgroup$
    – unicorn
    Commented Aug 18, 2020 at 7:56
  • $\begingroup$ @unicorn: To calculate P(data), one has to integrate P(data, parameters) over the parameters. This task is generally difficult. $\endgroup$
    – ocram
    Commented Aug 19, 2020 at 5:31
2
$\begingroup$

When applying Bayes' rule, we usually wish to infer the "parameters" and the "data" is already given. Thus, $\Pr(\textrm{data})$ is a constant and we can assume that it is just a normalizing factor.

Improve this answer
$\endgroup$
0
1
$\begingroup$

Most explanations of Bayes miss the mark. Consider the following for the role of Pr(B).

The crux of Bayes is the "update factor" $[Pr(B|A) / Pr(B)]$. This is the transformation applied to the prior.

If B always occurs in all states of the world, there is no information content & the update factor is 1.
In this case, $Pr(A|B) = Pr(A)$.

However, if B occurs frequently when A has occurred, but the overall probability of B occurring is very low, then there is high information content with respect to Pr(A).
The update factor will be HIGH and so $Pr(A|B) >> Pr(A)$.

For completeness, if B occurs rarely when A has occurred, but the overall probability of B occurring is very high, then there is also information content with respect to Pr(A), but in the opposite direction.
The update factor will be LOW and so $Pr(B|A) << Pr(A)$.

Purely mechanical explanations of Bayes seem to miss the genius of this simple equation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.