Under what circumstances can an improper prior be used in bayesian analysis?

Question

I am attempting to gain some intuition about the use of priors in bayesian analysis. I have read in some instances that an improper prior can be used when no information is known. However here is my first confusion, how exactly is an improper prior defined, I have seen some definitions that remark that it is a distribution that does not integrate to 1 and others affirm that it is a distribution that integrates to infinity, which one is it? Further how exactly does the prior distribution relate to the posterior, in terms of the attributes that the posterior must have when we use an improper prior. Is it the case that the posterior must be able to be integrated when we used an improper prior? must it be proper or improper? Further can an improper prior be used for valid bayesian inference?

Answer 1

The textbook definition is that an improper prior is a prior probability density that is not integrable, i.e. it does not have a finite integral. A CDF cannot be higher than 1 hence the name "improper". If a prior were integrable to a finite number, one could always divide the prior density to said number making it integrable to 1. Thus, it must be non-finite.

Using an improper prior may lead to an improper posterior, but it does not need to. For example, choosing the uniform prior over $\mathbb{R}$ as $\pi(\theta)=1 \ \forall \ \theta\in\mathbb{R}$ leads to it vanishing (multiplication by 1) in the posterior, thus integration of the likelihood times prior to get the norming constant $f(y)=\int_{-\infty}^{+\infty}f(y|\theta)\pi(\theta)d\theta$ is just integration over the likelihood. Even if this is not the case, for Bayesian simulation we only need the kernel of the posterior, i.e. the part that is proportional to the posterior, not the norming constant, i.e. $\pi(\theta|y) \propto f(y|\theta)\pi(\theta)$. Thus Integration to 1 is not needed for the posterior. The kernel is enough to simulate the posterior, which is most likely the case with Bayesian problems. Of course, if we 'know' the posterior, all inference regarding it is valid*, but some issues arise (see below).

*[EDIT: I've given this some thought and clearly you would not be able to calculate expectations, variances, credibility intervals in some cases of improper posteriors. I found this as an example of a simulation from such a posterior. There are obviously problems with them.]

---

[Extra content: Interest arises not whether or not improper priors can be used, but when looking at the issues that arise when using improper priors such as 1) the possibility of manipulation of Bayes factors and 2) marginalization paradoxes. Improper priors should best be avoided. Improper priors are used to model uninformativeness, i.e. the fact that we have no prior knowledge of the data. There are many proposals regarding uninformative (diffuse) priors, which need not be improper. To name a few prior proposals: Jeffrey's prior, Haldane's prior, Maximum Entropy prior, Maximal data information prior, empirical Bayes.]

Answer 2

To add a specific example to PaulG's +1 answer, consider the beta binomial setting.

Consider a prior $$ \theta \sim Beta\left( 0,1\right) $$ such that $$\pi \left( \theta \right) =\frac{\Gamma \left( 1\right)}{\Gamma\left(0\right)\Gamma\left(1\right) }\theta ^{-1}\left( 1-\theta \right) ^{1-1} =\frac{1}{\theta} $$ with $\int_0^1\frac{1}{\theta}d\theta=\ln\theta|^1_0=+\infty$. The posterior becomes $$ \pi(\theta|y) \propto \theta ^{k-1}\left( 1-\theta \right) ^{n-k}, $$ which is a $Beta\left( k,n-k+1\right)$-kernel. This is a proper posterior unless $k=0$. Similarly, $\theta \sim Beta\left( 1,0\right)$ leads to a $Beta\left( k+1,n-k\right)$-kernel, which is proper unless $k=n$. So, in this particular example, we are safe if $0<k<n$.

Improper priors are a more serious issue when using Bayes factors, and hence posterior odds ratios. As no normalization exists (this also addresses one of your questions - if the prior integrated to any other finite number $d$, we could renormalize by $1/d$ to have a prior that integrates to 1), an improper prior $\pi$ is equivalent to $c\pi$, $c>0$.

Thus we can write the Bayes factor as $$ \frac{\int f_{1}\left( y|\theta _{1}\right) c_1\pi _{1}\left(\theta_{1}\right) d\theta _{1}}{\int f_{2}\left( y|\theta _{2}\right) c_2\pi_{2}\left( \theta _{2}\right) d\theta _{2}} $$ Since the $c_i$ are arbitrary, so is the Bayes factor.