Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I know that priors need not be proper and that the likelihood function does not integrate to 1 either. But does the posterior need to be a proper distribution? What are the implications if it is/is not?

share|improve this question

4 Answers 4

up vote 1 down vote accepted

Any "distribution" must sum (or integrate) to 1. I can think a few examples where one might work with un-normalized distributions, but I am uncomfortable ever calling anything which marginalizes to anything but 1 a "distribution".

Given that you mentioned Bayesian posterior, I bet your question might come from a classification problem of searching for the optimal estimate of $x$ given some feature vector $d$

$$ \begin{align} \hat{x} &= \arg \max_x P_{X|D}(x|d) \\ &= \arg \max_x \frac{P_{D|X}(d|x) P_X(x)}{P_D(d)} \\ &= \arg \max_x {P_{D|X}(d|x) P_X(x)} \end{align} $$

where the last equality comes from the fact that $P_D$ doesn't depend on $x$. We can then choose our $\hat{x}$ exclusively based on the value $P_{D|X}(d|x) P_X(x)$ which is proportional to our Bayesian posterior, but do not confuse it for a probability!

share|improve this answer
    
@Zen would you mind being more explicit about what you think is wrong (or fundamentally incomplete) about this answer? –  whuber Mar 7 at 21:20
1  
One way to interpret the OP question "does the posterior need to be a proper distribution?" is to ask if it is mathematically possible to start with a proper prior and end with an improper posterior. Minka's answer gives an explicit example in which it happens. I tried to complement it with my answer and point out that this can only happen inside a set of zero prior predictive probability. –  Zen Mar 7 at 21:55
    
@Zen It seems to me that a closely related interpretation is "if the posterior is not proper, what information can I get from it?" This accepted answer looks like it provides useful and correct advice related to that in a special circumstance (which is clearly described). The acceptance looks to me like a signal that eretmochelys struck home with a shrewd guess about the circumstances. –  whuber Mar 7 at 22:03
    
OK. Thank you for your prompt assistance. –  Zen Mar 7 at 22:30

The posterior distribution need not be proper even if the prior is proper. For example, suppose $v$ has a Gamma prior with shape 0.25 (which is proper), and we model our datum $x$ as drawn from a Gaussian distribution with mean zero and variance $v$. Suppose $x$ is observed to be zero. Then the likelihood $p(x|v)$ is proportional to $v^{-0.5}$, which makes the posterior distribution for $v$ improper, since it is proportional to $v^{-1.25} e^{-v}$. This problem arises because of the wacky nature of continuous variables.

share|improve this answer
    
Cool example, Tom! –  Zen Mar 7 at 13:41
    
+1, though could you expand the answer to the OP's last sentence? Is this whacky posterior meaningful (can you do the kinds of things you'd usually do with a posterior), or is it more analogous to getting a NaN or Inf from some calculations? Is it a sign that something's wrong with your model? –  Wayne Mar 7 at 14:48
4  
There's nothing wrong with the model. This posterior is meaningful in the sense that if you receive another observation, you can multiply it in and possibly get back to a proper posterior. So it's not like a NaN, upon which all further operations are NaN. –  Tom Minka Mar 7 at 14:59

Defining the set $$ \text{Bogus Data} = \left\{ x:\int f(x\mid \theta)\,\pi(\theta)\,d\theta = \infty \right\} \, , $$ we have $$ \mathrm{Pr}\left(X\in\text{Bogus Data}\right) = \int_\text{Bogus Data} \int f(x\mid \theta)\,\pi(\theta)\,d\theta\,dx = \int_\text{Bogus Data} \infty\,dx \, . $$ The last integral will be equal to $\infty$ if the Lebesgue measure of $\text{Bogus Data}$ is positive. But this is impossible, because this integral gives you a probability (a real number between $0$ and $1$). Hence, it follows that the Lebesgue measure of $\text{Bogus Data}$ is equal to $0$, and, of course, it also follows that $\mathrm{Pr}\left(X\in\text{Bogus Data}\right)=0$.

In words: the prior predictive probability of those sample values that make the posterior improper is equal to zero.

Moral of the story: beware of null sets, they may bite, however improbable it may be.

share|improve this answer
2  
You once wrote: "If we can start with a proper prior and get an improper posterior, then I will quit inference." –  Tom Minka Mar 7 at 14:23
2  
A little bit tongue in cheek, there was an implicit quantifier: If we can start with a proper prior and get an improper posterior, for every possible sample value, then I will quit inference. ;-) –  Zen Mar 7 at 15:39
    
By the way, remarkable memory, Tom! –  Zen Mar 7 at 22:01

Improper posterior distribution only arises when you're having an improper prior distribution. The implication of this is that the asymptotic results do not hold. As an example, consider a binomial data consisting of $n$ success and 0 failures, if using $Beta(0,0)$ as the prior distribution, then the posterior will be improper. In this situation, the best is to think of a proper prior distribution to substitute your improper prior.

share|improve this answer
3  
This answer is incorrect. See my answer. –  Tom Minka Mar 7 at 13:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.