5
$\begingroup$

I am attempting to gain some intuition about the use of priors in bayesian analysis. I have read in some instances that an improper prior can be used when no information is known. However here is my first confusion, how exactly is an improper prior defined, I have seen some definitions that remark that it is a distribution that does not integrate to 1 and others affirm that it is a distribution that integrates to infinity, which one is it? Further how exactly does the prior distribution relate to the posterior, in terms of the attributes that the posterior must have when we use an improper prior. Is it the case that the posterior must be able to be integrated when we used an improper prior? must it be proper or improper? Further can an improper prior be used for valid bayesian inference?

$\endgroup$
2
  • 1
    $\begingroup$ If you take "integrate" in its generalised measure-theory meaning, then a proper distribution has integral $1$ across all possibilities, while an improper distribution has some other integral. If it is some other positive finite value, you can easily make the improper distribution proper. If it is zero then the improper distribution is useless. So the only interesting case is where it is infinite. $\endgroup$
    – Henry
    Nov 24 '20 at 2:20
  • 1
    $\begingroup$ An improper prior combined with the likelihood of the observations can produce a proper posterior distribution, and in that sense be useful for Bayesian inference. If in a particular case the posterior is also improper then it is less useful, similar to trying estimate the variance parameter of a Normal distribution from a single observation. $\endgroup$
    – Henry
    Nov 24 '20 at 2:26
7
$\begingroup$

The textbook definition is that an improper prior is a prior probability density that is not integrable, i.e. it does not have a finite integral. A CDF cannot be higher than 1 hence the name "improper". If a prior were integrable to a finite number, one could always divide the prior density to said number making it integrable to 1. Thus, it must be non-finite.

Using an improper prior may lead to an improper posterior, but it does not need to. For example, choosing the uniform prior over $\mathbb{R}$ as $\pi(\theta)=1 \ \forall \ \theta\in\mathbb{R}$ leads to it vanishing (multiplication by 1) in the posterior, thus integration of the likelihood times prior to get the norming constant $f(y)=\int_{-\infty}^{+\infty}f(y|\theta)\pi(\theta)d\theta$ is just integration over the likelihood. Even if this is not the case, for Bayesian simulation we only need the kernel of the posterior, i.e. the part that is proportional to the posterior, not the norming constant, i.e. $\pi(\theta|y) \propto f(y|\theta)\pi(\theta)$. Thus Integration to 1 is not needed for the posterior. The kernel is enough to simulate the posterior, which is most likely the case with Bayesian problems. Of course, if we 'know' the posterior, all inference regarding it is valid*, but some issues arise (see below).

*[EDIT: I've given this some thought and clearly you would not be able to calculate expectations, variances, credibility intervals in some cases of improper posteriors. I found this as an example of a simulation from such a posterior. There are obviously problems with them.]


[Extra content: Interest arises not whether or not improper priors can be used, but when looking at the issues that arise when using improper priors such as 1) the possibility of manipulation of Bayes factors and 2) marginalization paradoxes. Improper priors should best be avoided. Improper priors are used to model uninformativeness, i.e. the fact that we have no prior knowledge of the data. There are many proposals regarding uninformative (diffuse) priors, which need not be improper. To name a few prior proposals: Jeffrey's prior, Haldane's prior, Maximum Entropy prior, Maximal data information prior, empirical Bayes.]

$\endgroup$
2
  • 2
    $\begingroup$ The Haldane prior (Beta) is improper. Some Jeffreys priors are improper. $\endgroup$
    – Henry
    Nov 24 '20 at 2:01
  • $\begingroup$ An improper posterior distribution is not much use. For example with a Bernoulli parameter $\theta$ and a Haldane prior, and a single observation, the posterior probability distribution will be concentrated at $0$ or at $1$ depending on the observation, which suggests that all non-extreme possibilities will be almost entirely excluded; not a diffuse result $\endgroup$
    – Henry
    Nov 24 '20 at 2:05
4
$\begingroup$

To add a specific example to PaulG's +1 answer, consider the beta binomial setting.

Consider a prior $$ \theta \sim Beta\left( 0,1\right) $$ such that $$\pi \left( \theta \right) =\frac{\Gamma \left( 1\right)}{\Gamma\left(0\right)\Gamma\left(1\right) }\theta ^{-1}\left( 1-\theta \right) ^{1-1} =\frac{1}{\theta} $$ with $\int_0^1\frac{1}{\theta}d\theta=\ln\theta|^1_0=+\infty$. The posterior becomes $$ \pi(\theta|y) \propto \theta ^{k-1}\left( 1-\theta \right) ^{n-k}, $$ which is a $Beta\left( k,n-k+1\right)$-kernel. This is a proper posterior unless $k=0$. Similarly, $\theta \sim Beta\left( 1,0\right)$ leads to a $Beta\left( k+1,n-k\right)$-kernel, which is proper unless $k=n$. So, in this particular example, we are safe if $0<k<n$.

Improper priors are a more serious issue when using Bayes factors, and hence posterior odds ratios. As no normalization exists (this also addresses one of your questions - if the prior integrated to any other finite number $d$, we could renormalize by $1/d$ to have a prior that integrates to 1), an improper prior $\pi$ is equivalent to $c\pi$, $c>0$.

Thus we can write the Bayes factor as $$ \frac{\int f_{1}\left( y|\theta _{1}\right) c_1\pi _{1}\left(\theta_{1}\right) d\theta _{1}}{\int f_{2}\left( y|\theta _{2}\right) c_2\pi_{2}\left( \theta _{2}\right) d\theta _{2}} $$ Since the $c_i$ are arbitrary, so is the Bayes factor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.