# Why is the posterior distribution in Bayesian Inference often intractable?

I have a problem understanding why Bayesian Inference leads to intractable problems. The problem is often explained like this:

What I don't understand is why this integral has to be evaluated in the first place: It seems to me that the result of the integral is simply a normalization constant (as the dataset D is given). Why can one not simply calculate the posterior distribution as the numerator of the right-hand side and then infer this normalization constant by requiring that the integral over the posterior distribution has to be 1?

What am I missing?

Thanks!

• To whom it may concern: this question is squarely on-topic because it is about statistics.
– Sycorax
Apr 19, 2016 at 14:13
• The excerpt is poorly written. Be aware that $P(\mathcal D)$ is not the posterior distribution; it is the unconditional probability of the data (ie, irrespective of theta). Because $P(\mathcal D)$ will be the same for all models considered for the same dataset, it does not necessarily need to be computed. If you don't, you simply need to change the equals sign to 'proportional to' ($\propto$). Apr 19, 2016 at 14:38
• Could you provide the reference of that slide as I presume it was written by someone else? Apr 20, 2016 at 9:35
• The requirement to compute $p(\mathcal{D})$ only truly occurs when comparing models (this is sometimes called the evidence). When considering a single model, the numerator "suffices" to define the posterior. However, if you want to computed point estimators like posterior expectations or quantiles, you very quickly find you also need the denominator. Apr 20, 2016 at 9:38
• We are currently holding a workshop on normalising constants where you may find interesting entries for answering this question. Apr 21, 2016 at 10:58

## 2 Answers

Why can one not simply calculate the posterior distribution as the numerator of the right-hand side and then infer this normalization constant by requiring that the integral over the posterior distribution has to be 1?

This is precisely what is being done. The posterior distribution is $$P(\theta|D) = \dfrac{p(D|\theta) \, P(\theta)}{P(D)}.$$

The numerator on the right hand side is $P(D|\theta)P(\theta)$. This is a function over $\theta$ and to be a probability distribution, it has to integrate to 1. Thus we need to find the constant $c$, such that

\begin{align*} &\int_{\theta} cP(D|\theta) \, P(\theta)\, d\theta = 1\\ \Rightarrow & \int_{\theta} cP(D, \theta) \, d\theta = 1\\ \Rightarrow & cP(D) = 1\\ \Rightarrow& c = \dfrac{1}{P(D)}. \end{align*}

Thus, the normalizing constant is $P(D)$ which is often intractable, or overtly complicated.

• Another way to think about it is: which $\theta$ is best? You have to look at all of them! Jul 20, 2018 at 18:38
• @information_interchange What do you mean by "best"? Is there some optimization here? Jan 31, 2021 at 23:39

I had the same question. This great post explains it really well.

In a nutshell. It is intractable because the denominator has to evaluate the probability for ALL possible values of 𝜃; in most interesting cases ALL is a large amount. Whereas the numerator is for a single realization of 𝜃.

See Eqs. 4-8 in the post. Screenshot of the link:

• why does the proof you show here connect with the intractability question? Jan 30, 2021 at 20:56
• Please don't copy-paste images of text. Images take up more server space, but are needed for things like plots and diagrams. Rather, please typeset the essential elements of the text and math. May 12, 2022 at 16:59