Why *only* denominator(marginal likelihood) is difficult in Bayesian Inference, cant we keep track of it while calculating numerator

Question

First, this question might give impression that it is related to several question but other don't explain the confusion regarding nominator part.

For 2D, Given we have two parameters $\theta$ and $\phi$, we have double sum/integration in denominator like below

$$ \iint p(x,\theta,\phi) \,d\theta\,d\phi $$ $$\sum_{\theta=1}^{\infty} \sum_{\phi=1}^{\infty} p(x,\theta,\phi) $$

But since in numerator we compute numerator = likelihood*prior for all $\theta$ and $\phi$ as well, why can we keep track of them when doing that and add them in single operation at the end? or why numerator is not difficult as well?

(NOTE: It may be very stupid question because i think I am missing some thing major)

Answer 1

The numerator is typically easy to compute. I.e. $p(x | \theta, \phi)$ is just a likelihood function multiplied by a prior.

The problem here is integration. For a lot of functions, integration is straightforward. For example, $\int_0^1 x^2 \mathrm{d}x = \frac{1}{3}$. However, in general, it's not easy to compute an integral in an analytic form (i.e. I can't "write down" the answer).

I think the issue here is that we can compute $p(x|\theta, \phi)$ for lots of values of $\theta, \phi$ but a computer is probably going to do this for us. Although it feels like computers are arbitrarily precise, they do only work to finite precision. Therefore, this integration is going to be a numerical integration and thus approximate. As the number of parameters gets large, the quality of this approximation can deteriorate a lot.

When we are working with density function we expect $$\int_{\Phi} \int_{\Theta} \frac{p(x | \theta, \phi)}{\int_{\Phi} \int_{\Theta} p(x | u, v) \mathrm{d}u\, \mathrm{d}v} \mathrm{d}\theta\, \mathrm{d}\phi = 1$$

If we numerically approximate the denominator $\hat{I} = \frac{1}{N}\sum_{i = 1}^N p(x | \theta_i, \phi_i) $ then the approximate posterior $$\hat{\pi}(\theta, \phi |x) = \frac{p(x|\theta, \phi)}{\hat{I}}$$ will not be a valid density function therefore not a valid posterior.

And as I said, if we have more than two parameters, $\hat{I}$ could be a very poor approximation, so the approximate posterior (which isn't even a valid posterior!) could an awful estimate. Therefore, any conclusions you make from your posterior will be meaningless.

Answer 2

This problem has been extensively covered in this forum, to wit:

Why is computing the Bayesian Evidence difficult?

Bayesian MCMC methods that need to calculate the evidence / normalizing factor

Normalizing constant irrelevant in Bayes theorem?

Why Normalizing Factor is Required in Bayes Theorem?

What does it mean intuitively to know a pdf “up to a constant”?

Why is it necessary to sample from the posterior distribution if we already KNOW the posterior distribution?

and it is hard to see which aspect has not yet been sufficiently processed for the OP.

In short, computing the evidence or marginal likelihood is certainly NOT the only problem in Bayesian computation. When dealing with a single model, it is rarely necessary to compute this constant (and if need be there exist unbiased estimators of $I^{-1}$). When comparing different models, there exists a myriad of solutions, covered in the above answers. Numerical integration is very rarely a solution of relevance.