Does the Bayesian posterior need to be a proper distribution?

Question

I know that priors need not be proper and that the likelihood function does not integrate to 1 either. But does the posterior need to be a proper distribution? What are the implications if it is/is not?

Answer 1

(It is somewhat of a surprise to read the previous answers, which focus on the potential impropriety of the posterior when the prior is proper, since, as far as I can tell, the question is whether or not the posterior has to be proper (i.e., integrable to one) to be a proper (i.e., acceptable for Bayesian inference) posterior.)

In Bayesian statistics, the posterior distribution has to be a probability distribution, from which one can derive moments like the posterior mean $\mathbb{E}^\pi[h(\theta)|x]$ and probability statements like the coverage of a credible region, $\mathbb{P}(\pi(\theta|x)>\kappa|x)$. If $$\int f(x|\theta)\,\pi(\theta)\,\text{d}\theta = +\infty\,,\qquad (1)$$ the posterior $\pi(\theta|x)$ cannot be normalised into a probability density and Bayesian inference simply cannot be conducted. The posterior simply does not exist in such cases.

Actually, (1) must hold for all $x$'s in the sample space and not only for the observed $x$ for, otherwise, selecting the prior would depend on the data. This means that priors like Haldane's prior, $\pi(p)\propto \{1/p(1-p)\}$, on the probability $p$ of a Binomial or a Negative Binomial variable $X$ cannot be used, since the posterior is not defined for $x=0$.

I know of one exception when one can consider "improper posteriors": it is found in "The Art of Data Augmentation" by David van Dyk and Xiao-Li Meng. The improper measure is over a so-called working parameter $\alpha$ such that the observation is produced by the marginal of an augmented distribution $$f(x|\theta)=\int_{T(x^\text{aug})=x} f(x^\text{aug}|\theta,\alpha)\,\text{d}x^\text{aug}$$ and van Dyk and Meng put an improper prior $p(\alpha)$ on this working parameter $\alpha$ in order to speed up the simulation of $\pi(\theta|x)$ (which remains well-defined as a probability density) by MCMC.

In another perspective, somewhat related to the answer by eretmochelys, namely a perspective of Bayesian decision theory, a setting where (1) occurs could still be acceptable if it led to optimal decisions. Namely, if $L(\delta,\theta)\ge 0$ is a loss function evaluating the impact of using the decision $\delta$, a Bayesian optimal decision under the prior $\pi$ is given by $$\delta^\star(x)=\arg\min_\delta \int L(\delta,\theta) f(x|\theta)\,\pi(\theta)\,\text{d}\theta$$ and all that matters is that this integral is not everywhere (in $\delta$) infinite. Whether or not (1) holds is secondary for the derivation of $\delta^\star(x)$, even though properties like admissibility are only guaranteed when (1) holds.

Answer 2

The posterior distribution need not be proper even if the prior is proper. For example, suppose $v$ has a Gamma prior with shape 0.25 (which is proper), and we model our datum $x$ as drawn from a Gaussian distribution with mean zero and variance $v$. Suppose $x$ is observed to be zero. Then the likelihood $p(x|v)$ is proportional to $v^{-0.5}$, which makes the posterior distribution for $v$ improper, since it is proportional to $v^{-1.25} e^{-v}$. This problem arises because of the wacky nature of continuous variables.

Answer 3

Defining the set $$ \text{Bogus Data} = \left\{ x:\int f(x\mid \theta)\,\pi(\theta)\,d\theta = \infty \right\} \, , $$ we have $$ \mathrm{Pr}\left(X\in\text{Bogus Data}\right) = \int_\text{Bogus Data} \int f(x\mid \theta)\,\pi(\theta)\,d\theta\,dx = \int_\text{Bogus Data} \infty\,dx \, . $$ The last integral will be equal to $\infty$ if the Lebesgue measure of $\text{Bogus Data}$ is positive. But this is impossible, because this integral gives you a probability (a real number between $0$ and $1$). Hence, it follows that the Lebesgue measure of $\text{Bogus Data}$ is equal to $0$, and, of course, it also follows that $\mathrm{Pr}\left(X\in\text{Bogus Data}\right)=0$.

In words: the prior predictive probability of those sample values that make the posterior improper is equal to zero.

Moral of the story: beware of null sets, they may bite, however improbable it may be.

P.S. As pointed out by Prof. Robert in the comments, this reasoning blows up if the prior is improper.

Answer 4

Any "distribution" must sum (or integrate) to 1. I can think a few examples where one might work with un-normalized distributions, but I am uncomfortable ever calling anything which marginalizes to anything but 1 a "distribution".

Given that you mentioned Bayesian posterior, I bet your question might come from a classification problem of searching for the optimal estimate of $x$ given some feature vector $d$

$$ \begin{align} \hat{x} &= \arg \max_x P_{X|D}(x|d) \\ &= \arg \max_x \frac{P_{D|X}(d|x) P_X(x)}{P_D(d)} \\ &= \arg \max_x {P_{D|X}(d|x) P_X(x)} \end{align} $$

where the last equality comes from the fact that $P_D$ doesn't depend on $x$. We can then choose our $\hat{x}$ exclusively based on the value $P_{D|X}(d|x) P_X(x)$ which is proportional to our Bayesian posterior, but do not confuse it for a probability!

Answer 5

Later is better than never. Here is a natural and useful counterexample I believe, arising from Bayesian nonparametrics.

Suppose ${\mathbf{x}} = \left( {{x_1},...,{x_i},...{x_n}} \right) \in {\mathbb{R}^n}$ has posterior probability distribution

$p\left( {\left. {\mathbf{x}} \right|D} \right) \propto {e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}$

We want to evaluate the posterior expectation $\mathbb{E}\left. {\mathbf{x}} \right|D$. If ${\mathbf{A}}$ is positive definite, then let

$I \triangleq \int\limits_{{\mathbb{R}^n}} {{e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}{{\text{d}}^n}{\mathbf{x}}} = \sqrt {{{\left( {2\pi } \right)}^n}{{\left| {\mathbf{A}} \right|}^{ - 1}}} {e^{\frac{1}{2}{{\mathbf{J}}^{\mathbf{T}}}{{\mathbf{A}}^{ - 1}}{\mathbf{J}}}}$

By Leibniz rule/Feynman trick, we have

$ \frac{{\partial I}}{{\partial {J_j}}} = \int\limits_{{\mathbb{R}^n}} {\frac{{\partial {e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}}}{{\partial {J_j}}}{{\text{d}}^n}{\mathbf{x}}} = \int\limits_{\,{\mathbb{R}^n}} {{x_j}{e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}{{\text{d}}^n}{\mathbf{x}}} = \\ \frac{\partial }{{\partial {J_j}}}\sqrt {{{\left( {2\pi } \right)}^n}{{\left| {\mathbf{A}} \right|}^{ - 1}}} {e^{\frac{1}{2}{{\mathbf{J}}^{\mathbf{T}}}{{\mathbf{A}}^{ - 1}}{\mathbf{J}}}} = \sqrt {{{\left( {2\pi } \right)}^n}{{\left| {\mathbf{A}} \right|}^{ - 1}}} {e^{\frac{1}{2}{{\mathbf{J}}^{\mathbf{T}}}{{\mathbf{A}}^{ - 1}}{\mathbf{J}}}}\frac{\partial }{{\partial {J_j}}}\frac{1}{2}{{\mathbf{J}}^{\mathbf{T}}}{{\mathbf{A}}^{ - 1}}{\mathbf{J}} = \\ \frac{1}{2}I\frac{\partial }{{\partial {J_j}}}{{\mathbf{J}}^{\mathbf{T}}}{{\mathbf{A}}^{ - 1}}{\mathbf{J}} = I\sum\limits_{i = 1}^n {{\mathbf{A}}_{ij}^{ - 1}{{\mathbf{J}}_i}} \\ $

Therefore

$\mathbb{E}\left. {{x_j}} \right|D = \frac{{\int\limits_{\,{\mathbb{R}^n}} {{x_j}{e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}{{\text{d}}^n}{\mathbf{x}}} }}{{\int\limits_{\,{\mathbb{R}^n}} {{e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}} + {{\mathbf{J}}^{\mathbf{T}}}{\mathbf{x}}}}{{\text{d}}^n}{\mathbf{x}}} }} = \sum\limits_{i = 1}^n {{\mathbf{A}}_{ij}^{ - 1}{{\mathbf{J}}_i}} $

and

$\mathbb{E}\left. {\mathbf{x}} \right|D = {{\mathbf{A}}^{ - 1}}{\mathbf{J}}$

Now, if ${\mathbf{A}}$ is only positive semi-definite and singular, so that $p\left( {\left. {\mathbf{x}} \right|D} \right)$ is improper, degenerate and

$\int\limits_{{\mathbb{R}^n}} {p\left( {\left. {\mathbf{x}} \right|D} \right){{\text{d}}^n}{\mathbf{x}}} = + \infty $

it suffices to replace the matrix inverse ${{\mathbf{A}}^{ - 1}}$ by its Moore-Penrose pseudoinverse ${{\mathbf{A}}^ + }$ to get

$\mathbb{E}\left. {\mathbf{x}} \right|D = {{\mathbf{A}}^ + }{\mathbf{J}}$

IT WORKS. Same for higher moments. So, it seems that a Bayesian posterior does not need to be proper/non-degenerate in order to be proper, that is to yield legitimate and useful inferences.

Answer 6

Improper posterior distribution only arises when you're having an improper prior distribution. The implication of this is that the asymptotic results do not hold. As an example, consider a binomial data consisting of $n$ success and 0 failures, if using $Beta(0,0)$ as the prior distribution, then the posterior will be improper. In this situation, the best is to think of a proper prior distribution to substitute your improper prior.