How can an improper prior lead to a proper posterior distribution?

Question

We know that in the case of a proper prior distribution,

$P(\theta \mid X) = \dfrac{P(X \mid \theta)P(\theta)}{P(X)}$

$ \propto P(X \mid \theta)P(\theta)$.

The usual justification for this step is that the marginal distribution of $X$, $P(X)$, is constant with respect to $\theta$ and can thus be ignored when deriving the posterior distribution.

However, in the case of an improper prior, how do you know that the posterior distribution actually exists? There seems to be something missing in this seemingly circular argument. In other words, if I assume the posterior exists, I understand the mechanics of how to derive the posterior, but I seem to be missing the theoretical justification for why it even exists.

P.S. I also recognize that there are cases in which an improper prior leads to an improper posterior.

Answer 1

We generally accept posteriors from improper priors $\pi(\theta)$ if $$ \frac{\pi(X \mid \theta) \pi(\theta)}{\pi(X)} $$ exists and is a valid probability distribution (i.e., it integrates exactly to 1 over the support). Essentially this boils down to $\pi(X) = \int \pi(X \mid \theta) \pi(\theta) \,d\theta$ being finite. If this is the case, then we call this quantity $\pi(\theta \mid X)$ and accept it as the posterior distribution that we want. However, it is important to note that this is NOT a posterior distribution, nor is it a conditional probability distribution (these two terms are synonymous in the context here).

Now, I said we accept 'posterior' distributions from improper priors given the above. The reason they are accepted is because the prior $\pi(\theta)$ will still give us relative 'scores' on the parameter space; i.e., the ratio $\frac{\pi(\theta_1)}{\pi(\theta_2)}$ brings meaning to our analysis. The meaning we get from improper priors in some cases may not be available in proper priors. This is a potential justification for using them. See Sergio's answer for a more thorough examination of the practical motivation for improper priors.

It's worth noting that this quantity $\pi(\theta \mid X)$ does have desirable theoretical properties as well, Degroot & Schervish:

Improper priors are not true probability distributions, but if we pretend that they are, we will compute posterior distributions that approximate the posteriors that we would have obtained using proper conjugate priors with extreme values of the prior hyperparameters.

Answer 2

There are a "theoretical" answer and a "pragmatic" one.

From a theroretical point of view, when a prior is improper the posterior does not exist (well, look at Matthew's answer for a sounder statement), but may be approximated by a limiting form.

If the data comprise a conditionally i.i.d. sample from the Bernoulli distribution with parameter $\theta$, and $\theta$ has the beta distribution with parameters $\alpha$ and $\beta$, the posterior distribution of $\theta$ is the beta distribution with parameters $\alpha + s, \beta+n-s$ ($n$ observations, $s$ successes) and its mean is $(\alpha+s)/(\alpha+\beta+n)$. If we use the improper (and unreal) beta distribution prior with prior hypeparameters $\alpha=\beta=0$, and pretend that $\pi(\theta)\propto\theta^{-1}(1-\theta)^{-1}$, we obtain a proper posterior proportional to $\theta^{s-1}(1-\theta)^{n-s-1}$, i.e. the p.d.f. of the beta distribution with parameters $s$ and $n-s$ except for a constant factor. This is the limiting form of the posterior for a beta prior with parameters $\alpha\to 0$ and $\beta\to 0$ (Degroot & Schervish, Example 7.3.13).

In a normal model with mean $\theta$, known variance $\sigma^2$, and a $\mathcal{N}(\mu_0,\tau^2_0)$ prior distribution for $\theta$, if the prior precision, $1/\tau^2_0$, is small relative to the data precision, $n/\sigma^2$, then the posterior distribution is approximately as if $\tau^2_0=\infty$: $$p(\theta\mid x)\approx \mathcal{N}(\theta\mid\bar{x},\sigma^2/n)$$ i.e. the posterior distribution is approximately that which would result from assuming $p(\theta)$ is proportional to a constant for $\theta\in(-\infty,\infty)$, a distribution that is not strictly possible, but the limiting form of the posterior as $\tau^2_0$ approaches $\infty$ does exist (Gelman et al., p. 52).

From a "pragmatic" point of view, $p(x\mid\theta)p(\theta)=0$ when $p(x\mid\theta)=0$ whatever $p(\theta)$ is, so if $p(x\mid\theta)\ne 0$ in $(a,b)$, then $\int_{-\infty}^{\infty}p(x\mid\theta)p(\theta)d\theta=\int_a^b p(x\mid\theta)p(\theta)d\theta$. Improper priors may be employed to represent the local behavior of the prior distribution in the region where the likelihood is appreciable, say $(a,b)$. By supposing that to a sufficient approximation a prior follows forms such as $f(x)=k, x\in(-\infty,\infty)$ or $f(x)=kx^{-1}, x\in(0,\infty)$ only over $(a,b)$, that it suitably tails to zero outside that range, we ensure the priors actually used are proper (Box and Tiao, p. 21). So if the prior distribution of $\theta$ is $\mathcal{U}(-\infty,\infty)$ but $(a,b)$ is bounded, it is as if $\theta\sim\mathcal{U}(a,b)$, i.e. $p(x\mid\theta)p(\theta)=p(x\mid\theta)k\propto p(x\mid\theta)$. For a concrete example, this is what happens in Stan: if no prior is specified for a parameter, it is implicitly given a uniform prior on its support and this is handled as a multiplication of the likelihood by a constant.

Answer 3

However, in the case of an improper prior, how do you know that the posterior distribution actually exists?

The posterior might not be proper either. If the prior is improper and the likelihood is flat (because there are no meaningful observations), then the posterior equals the prior and is also improper.

Usually you have some observations, and usually the likelihood is not flat, so the posterior is proper.