3
$\begingroup$

First, this question might give impression that it is related to several question but other don't explain the confusion regarding nominator part.

For 2D, Given we have two parameters $\theta$ and $\phi$, we have double sum/integration in denominator like below

$$ \iint p(x,\theta,\phi) \,d\theta\,d\phi $$ $$\sum_{\theta=1}^{\infty} \sum_{\phi=1}^{\infty} p(x,\theta,\phi) $$

But since in numerator we compute numerator = likelihood*prior for all $\theta$ and $\phi$ as well, why can we keep track of them when doing that and add them in single operation at the end? or why numerator is not difficult as well?

(NOTE: It may be very stupid question because i think I am missing some thing major)

$\endgroup$
5
  • $\begingroup$ (not sure) but wont likelihood be computer for all collection if you need posterior for all hypothesis? $\endgroup$
    – A.B
    Feb 15 at 10:22
  • $\begingroup$ What do you mean by "all collection"? $\endgroup$
    – Xi'an
    Feb 15 at 10:33
  • $\begingroup$ I mean for all values of $\theta$ and $\phi$ $\endgroup$
    – A.B
    Feb 15 at 10:37
  • $\begingroup$ for exampe if we have values 0.1 to 0.9 for both, each value of both these paramaters can be cosidered one hypothesis. but likilihood would be different for each value, when we calculate that likelihood for each value(combination), why cant we just store it somewhere and use it to divide as normalising constant (may be my question isnt right) $\endgroup$
    – A.B
    Feb 15 at 10:40
  • 1
    $\begingroup$ Monte Carlo methods are provably more efficient than a sheer (blind) numerical integration in most cases of interest. $\endgroup$
    – Xi'an
    Feb 15 at 10:42
4
$\begingroup$

The numerator is typically easy to compute. I.e. $p(x | \theta, \phi)$ is just a likelihood function multiplied by a prior.

The problem here is integration. For a lot of functions, integration is straightforward. For example, $\int_0^1 x^2 \mathrm{d}x = \frac{1}{3}$. However, in general, it's not easy to compute an integral in an analytic form (i.e. I can't "write down" the answer).

I think the issue here is that we can compute $p(x|\theta, \phi)$ for lots of values of $\theta, \phi$ but a computer is probably going to do this for us. Although it feels like computers are arbitrarily precise, they do only work to finite precision. Therefore, this integration is going to be a numerical integration and thus approximate. As the number of parameters gets large, the quality of this approximation can deteriorate a lot.

When we are working with density function we expect $$\int_{\Phi} \int_{\Theta} \frac{p(x | \theta, \phi)}{\int_{\Phi} \int_{\Theta} p(x | u, v) \mathrm{d}u\, \mathrm{d}v} \mathrm{d}\theta\, \mathrm{d}\phi = 1$$

If we numerically approximate the denominator $\hat{I} = \frac{1}{N}\sum_{i = 1}^N p(x | \theta_i, \phi_i) $ then the approximate posterior $$\hat{\pi}(\theta, \phi |x) = \frac{p(x|\theta, \phi)}{\hat{I}}$$ will not be a valid density function therefore not a valid posterior.

And as I said, if we have more than two parameters, $\hat{I}$ could be a very poor approximation, so the approximate posterior (which isn't even a valid posterior!) could an awful estimate. Therefore, any conclusions you make from your posterior will be meaningless.

$\endgroup$
8
  • 1
    $\begingroup$ $\hat I$ is not a converging approximation of the denominator. Even considering $\hat I/N$, it depends on the choice of the $(\theta_i,\phi_i)$'s. $\endgroup$
    – Xi'an
    Feb 15 at 10:19
  • $\begingroup$ Added in $1/N$ term to $\hat{I}$ as a "naive"/simple approximation of the denominator - thanks for flagging $\endgroup$
    – jcken
    Feb 15 at 10:24
  • $\begingroup$ thankyou for the answer, for likelihood, we still need to caclilate it for all hypothesis/parameters, right? then why is it easy to compute? (sorry if i am not very clear) $\endgroup$
    – A.B
    Feb 15 at 10:24
  • $\begingroup$ What are your conditions for $\hat I$ to converge? $\endgroup$
    – Xi'an
    Feb 15 at 10:33
  • $\begingroup$ A.B. - we simply want to evaluate the likelihood at any parameter whereas to integrate "exactly" via an infinite sum we would need to calculate at all values. Xi'an - no conditions in particular. I guess I am simply assuming $\hat{I}$ converges as $N \to \infty$ and $\phi_i, \theta_i$ are somehow "sensible", e.g. grid points or a sobol sequence $\endgroup$
    – jcken
    Feb 15 at 10:54
8
$\begingroup$

This problem has been extensively covered in this forum, to wit:

Why is computing the Bayesian Evidence difficult?

Bayesian MCMC methods that need to calculate the evidence / normalizing factor

Normalizing constant irrelevant in Bayes theorem?

Why Normalizing Factor is Required in Bayes Theorem?

What does it mean intuitively to know a pdf “up to a constant”?

Why is it necessary to sample from the posterior distribution if we already KNOW the posterior distribution?

and it is hard to see which aspect has not yet been sufficiently processed for the OP.

In short, computing the evidence or marginal likelihood is certainly NOT the only problem in Bayesian computation. When dealing with a single model, it is rarely necessary to compute this constant (and if need be there exist unbiased estimators of $I^{-1}$). When comparing different models, there exists a myriad of solutions, covered in the above answers. Numerical integration is very rarely a solution of relevance.

$\endgroup$
1
  • $\begingroup$ thankyou for the answer, i have tried to convey my confision in the comment to original question $\endgroup$
    – A.B
    Feb 15 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.