Does the Bayesian posterior need to be a proper distribution?

I know that priors need not be proper and that the likelihood function does not integrate to 1 either. But does the posterior need to be a proper distribution? What are the implications if it is/is not?

-

The posterior distribution need not be proper even if the prior is proper. For example, suppose $v$ has a Gamma prior with shape 0.25 (which is proper), and we model our datum $x$ as drawn from a Gaussian distribution with mean zero and variance $v$. Suppose $x$ is observed to be zero. Then the likelihood $p(x|v)$ is proportional to $v^{-0.5}$, which makes the posterior distribution for $v$ improper, since it is proportional to $v^{-1.25} e^{-v}$. This problem arises because of the wacky nature of continuous variables.

-
Cool example, Tom! –  Zen Mar 7 at 13:41
+1, though could you expand the answer to the OP's last sentence? Is this whacky posterior meaningful (can you do the kinds of things you'd usually do with a posterior), or is it more analogous to getting a NaN or Inf from some calculations? Is it a sign that something's wrong with your model? –  Wayne Mar 7 at 14:48
There's nothing wrong with the model. This posterior is meaningful in the sense that if you receive another observation, you can multiply it in and possibly get back to a proper posterior. So it's not like a NaN, upon which all further operations are NaN. –  Tom Minka Mar 7 at 14:59

Defining the set $$\text{Bogus Data} = \left\{ x:\int f(x\mid \theta)\,\pi(\theta)\,d\theta = \infty \right\} \, ,$$ we have $$\mathrm{Pr}\left(X\in\text{Bogus Data}\right) = \int_\text{Bogus Data} \int f(x\mid \theta)\,\pi(\theta)\,d\theta\,dx = \int_\text{Bogus Data} \infty\,dx \, .$$ The last integral will be equal to $\infty$ if the Lebesgue measure of $\text{Bogus Data}$ is positive. But this is impossible, because this integral gives you a probability (a real number between $0$ and $1$). Hence, it follows that the Lebesgue measure of $\text{Bogus Data}$ is equal to $0$, and, of course, it also follows that $\mathrm{Pr}\left(X\in\text{Bogus Data}\right)=0$.

In words: the prior predictive probability of those sample values that make the posterior improper is equal to zero.

Moral of the story: beware of null sets, they may bite, however improbable it may be.

-
You once wrote: "If we can start with a proper prior and get an improper posterior, then I will quit inference." –  Tom Minka Mar 7 at 14:23
A little bit tongue in cheek, there was an implicit quantifier: If we can start with a proper prior and get an improper posterior, for every possible sample value, then I will quit inference. ;-) –  Zen Mar 7 at 15:39
By the way, remarkable memory, Tom! –  Zen Mar 7 at 22:01

Any "distribution" must sum (or integrate) to 1. I can think a few examples where one might work with un-normalized distributions, but I am uncomfortable ever calling anything which marginalizes to anything but 1 a "distribution".

Given that you mentioned Bayesian posterior, I bet your question might come from a classification problem of searching for the optimal estimate of $x$ given some feature vector $d$

\begin{align} \hat{x} &= \arg \max_x P_{X|D}(x|d) \\ &= \arg \max_x \frac{P_{D|X}(d|x) P_X(x)}{P_D(d)} \\ &= \arg \max_x {P_{D|X}(d|x) P_X(x)} \end{align}

where the last equality comes from the fact that $P_D$ doesn't depend on $x$. We can then choose our $\hat{x}$ exclusively based on the value $P_{D|X}(d|x) P_X(x)$ which is proportional to our Bayesian posterior, but do not confuse it for a probability!

-
Improper posterior distribution only arises when you're having an improper prior distribution. The implication of this is that the asymptotic results do not hold. As an example, consider a binomial data consisting of $n$ success and 0 failures, if using $Beta(0,0)$ as the prior distribution, then the posterior will be improper. In this situation, the best is to think of a proper prior distribution to substitute your improper prior.