# How can an improper prior lead to a proper posterior distribution?

We know that in the case of a proper prior distribution,

$P(\theta \mid X) = \dfrac{P(X \mid \theta)P(\theta)}{P(X)}$

$\propto P(X \mid \theta)P(\theta)$.

The usual justification for this step is that the marginal distribution of $X$, $P(X)$, is constant with respect to $\theta$ and can thus be ignored when deriving the posterior distribution.

However, in the case of an improper prior, how do you know that the posterior distribution actually exists? There seems to be something missing in this seemingly circular argument. In other words, if I assume the posterior exists, I understand the mechanics of how to derive the posterior, but I seem to be missing the theoretical justification for why it even exists.

P.S. I also recognize that there are cases in which an improper prior leads to an improper posterior.

-

We generally accept posteriors from improper priors $\pi(\theta)$ if $$\frac{\pi(X \mid \theta) \pi(\theta)}{\pi(X)}$$ exists and is a valid probability distribution (i.e., it integrates exactly to 1 over the support). Essentially this boils down to $\pi(X) = \int \pi(X \mid \theta) \pi(\theta) \,d\theta$ being finite. If this is the case, then we call this quantity $\pi(\theta \mid X)$ and accept it as the posterior distribution that we want. However, it is important to note that this is NOT a posterior distribution, nor is it a conditional probability distribution (these two terms are synonymous in the context here).

Now, I said we accept 'posterior' distributions from improper priors given the above. The reason they are accepted is because the prior $\pi(\theta)$ will still give us relative 'scores' on the parameter space; i.e., the ratio $\frac{\pi(\theta_1)}{\pi(\theta_2)}$ brings meaning to our analysis. The meaning we get from improper priors in some cases may not be available in proper priors. This is a potential justification for using them. See Sergio's answer for a more thorough examination of the practical motivation for improper priors.

It's worth noting that this quantity $\pi(\theta \mid X)$ does have desirable theoretical properties as well, Degroot & Schervish:

Improper priors are not true probability distributions, but if we pretend that they are, we will compute posterior distributions that approximate the posteriors that we would have obtained using proper conjugate priors with extreme values of the prior hyperparameters.

-
I am confused by a few things in your answer. You say that we accept posteriors if the above is finite. Does that mean if that integral is not finite the posterior will not be finite? Also, you seem to imply that we use the posterior in this case, but it is not a real distribution- is that right? are there not cases where it is a real distribution? Also, what does the ratio of priors have to do with this? I don't see the connection. –  Ben Elizabeth Ward May 15 '14 at 3:35
@BenElizabethWard If $\pi(\theta \mid X)$ exists, then the integral $\pi(X)$ must exist (and thus be finite). The contrapositive is true as well: if $\pi(X)$ does not exist (is infinite), then $\pi(\theta \mid X)$ does not exist. When it it exists and is a valid probability distribution, $\pi(\theta \mid X)$ is a probability distribution. However, it is not a posterior distribution for $\pi(\theta)$ with the given data likelihood $\pi(X \mid \theta)$. The posterior for that prior does not exist. We accept $\pi(\theta \mid X)$ in our analysis because it is an approximation. –  Matt May 15 '14 at 3:44
@BenElizabethWard The ratio was used to demonstrate that the prior still contains useful information that we might not be able to load into a proper prior. I will edit my answer to include this. –  Matt May 15 '14 at 3:45
@jsk $\pi(\theta)$ is not a probability distribution, but the definition of posterior distribution requires that $\pi(\theta)$ be a probability distribution, so it's cheating to call $\pi(\theta \mid X)$ a posterior distribution when it is a probability distribution. Degroot & Schervish say '..we will compute posterior distributions that..' by which they're assuming you agreed to 'pretend that they [the improper priors] are [proper priors]' as expressed earlier in the quote. –  Matt May 15 '14 at 4:23

There are a "theoretical" answer and a "pragmatic" one.

From a theroretical point of view, when a prior is improper the posterior does not exist (well, look at Matthew's answer for a sounder statement), but may be approximated by a limiting form.

If the data comprise a conditionally i.i.d. sample from the Bernoulli distribution with parameter $\theta$, and $\theta$ has the beta distribution with parameters $\alpha$ and $\beta$, the posterior distribution of $\theta$ is the beta distribution with parameters $\alpha + s, \beta+n-s$ ($n$ observations, $s$ successes) and its mean is $(\alpha+s)/(\alpha+\beta+n)$. If we use the improper (and unreal) beta distribution prior with prior hypeparameters $\alpha=\beta=0$, and pretend that $\pi(\theta)\propto\theta^{-1}(1-\theta)^{-1}$, we obtain a proper posterior proportional to $\theta^{s-1}(1-\theta)^{n-s-1}$, i.e. the p.d.f. of the beta distribution with parameters $s$ and $n-s$ except for a constant factor. This is the limiting form of the posterior for a beta prior with parameters $\alpha\to 0$ and $\beta\to 0$ (Degroot & Schervish, Example 7.3.13).

In a normal model with mean $\theta$, known variance $\sigma^2$, and a $\mathcal{N}(\mu_0,\tau^2_0)$ prior distribution for $\theta$, if the prior precision, $1/\tau^2_0$, is small relative to the data precision, $n/\sigma^2$, then the posterior distribution is approximately as if $\tau^2_0=\infty$: $$p(\theta\mid x)\approx \mathcal{N}(\theta\mid\bar{x},\sigma^2/n)$$ i.e. the posterior distribution is approximately that which would result from assuming $p(\theta)$ is proportional to a constant for $\theta\in(-\infty,\infty)$, a distribution that is not strictly possible, but the limiting form of the posterior as $\tau^2_0$ approaches $\infty$ does exist (Gelman et al., p. 52).

From a "pragmatic" point of view, $p(x\mid\theta)p(\theta)=0$ when $p(x\mid\theta)=0$ whatever $p(\theta)$ is, so if $p(x\mid\theta)\ne 0$ in $(a,b)$, then $\int_{-\infty}^{\infty}p(x\mid\theta)p(\theta)d\theta=\int_a^b p(x\mid\theta)p(\theta)d\theta$. Improper priors may be employed to represent the local behavior of the prior distribution in the region where the likelihood is appreciable, say $(a,b)$. By supposing that to a sufficient approximation a prior follows forms such as $f(x)=k, x\in(-\infty,\infty)$ or $f(x)=kx^{-1}, x\in(0,\infty)$ only over $(a,b)$, that it suitably tails to zero outside that range, we ensure the priors actually used are proper (Box and Tiao, p. 21). So if the prior distribution of $\theta$ is $\mathcal{U}(-\infty,\infty)$ but $(a,b)$ is bounded, it is as if $\theta\sim\mathcal{U}(a,b)$, i.e. $p(x\mid\theta)p(\theta)=p(x\mid\theta)k\propto p(x\mid\theta)$. For a concrete example, this is what happens in Stan: if no prior is specified for a parameter, it is implicitly given a uniform prior on its support and this is handled as a multiplication of the likelihood by a constant.

-
Can you say more about why it does not exist from a theoretical point of view? –  jsk May 15 '14 at 16:29
I couldn't expound better than Matthew in his answer and in his comments. –  Sergio May 15 '14 at 16:38
In the pragmatic section, what is y? Also in that section, should some of the $p(\theta \mid x)$ terms be the likelihood $p(x\mid \theta)$? –  jsk May 15 '14 at 16:41
Oops! Fixed (I hope), thanks! –  Sergio May 15 '14 at 17:02
Thanks. I Think there might be one more mistake... You write $P(\theta)=kx^{-1}$, but the prior can't depend on $x$. Do you mean $P(\theta)=k\theta^{-1}$? –  jsk May 15 '14 at 17:11

However, in the case of an improper prior, how do you know that the posterior distribution actually exists?

The posterior might not be proper either. If the prior is improper and the likelihood is flat (because there are no meaningful observations), then the posterior equals the prior and is also improper.

Usually you have some observations, and usually the likelihood is not flat, so the posterior is proper.

-