15
$\begingroup$

I have a problem understanding why Bayesian Inference leads to intractable problems. The problem is often explained like this:

enter image description here

What I don't understand is why this integral has to be evaluated in the first place: It seems to me that the result of the integral is simply a normalization constant (as the dataset D is given). Why can one not simply calculate the posterior distribution as the numerator of the right-hand side and then infer this normalization constant by requiring that the integral over the posterior distribution has to be 1?

What am I missing?

Thanks!

$\endgroup$
  • 6
    $\begingroup$ To whom it may concern: this question is squarely on-topic because it is about statistics. $\endgroup$ – Sycorax Apr 19 '16 at 14:13
  • 3
    $\begingroup$ The excerpt is poorly written. Be aware that $P(\mathcal D)$ is not the posterior distribution; it is the unconditional probability of the data (ie, irrespective of theta). Because $P(\mathcal D)$ will be the same for all models considered for the same dataset, it does not necessarily need to be computed. If you don't, you simply need to change the equals sign to 'proportional to' ($\propto$). $\endgroup$ – gung Apr 19 '16 at 14:38
  • $\begingroup$ Could you provide the reference of that slide as I presume it was written by someone else? $\endgroup$ – Xi'an Apr 20 '16 at 9:35
  • 3
    $\begingroup$ The requirement to compute $p(\mathcal{D})$ only truly occurs when comparing models (this is sometimes called the evidence). When considering a single model, the numerator "suffices" to define the posterior. However, if you want to computed point estimators like posterior expectations or quantiles, you very quickly find you also need the denominator. $\endgroup$ – Xi'an Apr 20 '16 at 9:38
  • $\begingroup$ We are currently holding a workshop on normalising constants where you may find interesting entries for answering this question. $\endgroup$ – Xi'an Apr 21 '16 at 10:58
7
$\begingroup$

Why can one not simply calculate the posterior distribution as the numerator of the right-hand side and then infer this normalization constant by requiring that the integral over the posterior distribution has to be 1?

This is precisely what is being done. The posterior distribution is $$P(\theta|D) = \dfrac{p(D|\theta) \, P(\theta)}{P(D)}. $$

The numerator on the right hand side is $P(D|\theta)P(\theta)$. This is a function over $\theta$ and to be a probability distribution, it has to integrate to 1. Thus we need to find the constant $c$, such that

\begin{align*} &\int_{\theta} cP(D|\theta) \, P(\theta)\, d\theta = 1\\ \Rightarrow & \int_{\theta} cP(D, \theta) \, d\theta = 1\\ \Rightarrow & cP(D) = 1\\ \Rightarrow& c = \dfrac{1}{P(D)}. \end{align*}

Thus, the normalizing constant is $P(D)$ which is often intractable, or overtly complicated.

$\endgroup$
  • $\begingroup$ Another way to think about it is: which $\theta$ is best? You have to look at all of them! $\endgroup$ – information_interchange Jul 20 '18 at 18:38
1
$\begingroup$

I had the same question. This great post explains it really well.

In a nutshell. It is intractable because the denominator has to evaluate the probability for ALL possible values of 𝜃; in most interesting cases ALL is a large amount. Whereas the numerator is for a single realization of 𝜃.

See Eqs. 4-8 in the post. Screenshot of the link:

Here is an screenshot of the link

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.