5
$\begingroup$

In Bayesian estimation, we need to compute the normalizing factor P(X). Say that our parameter space was y. Then in order to compute the Bayesian evidence we'd need to marginalize the evidence over all the possible parameters in our parameter space:

enter image description here

I don't understand why that is so difficult to compute. Is that not a straightforward application of the total law of probability?

$\endgroup$
1
3
$\begingroup$

I like @ShijiaBian's answer. I would add the following.

The normalizing constant is important because without it, (1) you won't have a valid probability distribution and (2) you can't assess relative probabilities of values of the parameter. For example, if you modeled data $x_t$ as Gaussian conditional on a mean $\theta$ that was modeled as Poisson, you would not be able to average the likelihood over the values of the parameter because the infinite sum over the product of the two PDFs' kernels is not available in closed form. Mathematically:

$$ \begin{align} p(\theta) &= \text{Poisson}(\lambda)\\ p(x_t|\theta) &= \mathcal{N}(\theta, \sigma^2)\\ p(\theta | \mathbf{X}) &= \frac{p(\theta)\prod_tp(x_t|\theta)}{\sum_\Theta p(\theta)\prod_tp(x_t|\theta)} \end{align} $$

Expanding the numerator, you'll find that:

$$ p(\theta | \mathbf{X}) \propto \frac{1}{\theta!}\lambda^\theta(2\pi\sigma^2)^{-T/2}\prod_t\exp{\left[\frac{-1}{2\sigma^2}(x_t^2 - 2\theta x_t + \theta^2) - \frac{\lambda}{T}\right]} $$

To normalize this function, you'd have to sum over all the possible (discrete) values of $\theta$: $0, 1, 2, \ldots, \infty$. This is impossible analytically because there is no closed-form expression for an infinite sum of the above form. If you don't do this, however, your function will not integrate to $1$ and you won't have a valid probability density. Furthermore, normalizing ensures that for each value of $\theta = \theta^*$, you can exactly determine the relative probability of $\theta^*$ relative to other values of $\theta$.

Expanding on this second point, if you only normalized for values of $\theta$, say, from $0$ through $10$, then you cannot compare how likely values of $\theta$ outside that range to values inside that range. This does suggest, however, that if you have some belief about the range of values for which $\theta$ may be restricted, you could truncate your distribution to that range and perform the summation numerically within that range, like $0$ to $10$. Then, you would have a valid probability distribution (a truncated Poisson) over the range of values from $0$ to $10$. This is much harder, however, when $\theta$ is continuous (say, Beta or Gamma distributed), although you could perform numerical integration. Numerical integration is difficult in high dimensions, however, so you'd have to restrict the dimension of $\theta$ to something that is computationally feasible.

$\endgroup$
1
  • $\begingroup$ does $P(\theta|X)$ denotes posterior for one value of $\theta$ or all? you say that To normalize this function, you'd have to sum over all the possible (discrete) values of θ: 0,1,2,…,∞, but for calculating $P(\theta|X)$, wont we need to calculate likelihood*prior for all values of $\theta$ in nominator as well? $\endgroup$
    – A.B
    Feb 24 at 20:38
3
$\begingroup$

The posterior distribution has no relationship with the law of total probability, even though they are similar looking.

The given $P_X(x)$ is the normalizing constant. The reason that this is hard to compute is because 1). the conjugate property only can be applied for some specific distributions; 2). The prior and the likelihood function can be high dimensional, which is very difficult to integrate; 3). The integral might not be closed form.

This is the reason why the resampling method has to play a role in Bayesian approximation.

$\endgroup$
2
  • $\begingroup$ Thanks for the response. I sorta knew all those characteristics. I have seen full solutions for computing the prior of the evidence, and all they seem to do is add the probabilities of all parameter values up together. This is where I get the total law from, it just says that we can compute the marginal by summing up all probabilities $\endgroup$
    – user46925
    Mar 22 '16 at 2:43
  • $\begingroup$ The law of total probability relates conditional and marginal distributions to the joint distribution; it states that the marginal distribution of X is the average of p(X | Y) weighted by p(Y). This is used to get the normalizing constant. $\endgroup$ Apr 27 '16 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy