Is a vague prior the same as a non-informative prior?

Question

This is a question about terminology. Is a "vague prior" the same as a non-informative prior, or is there some difference between the two? My impression is that they are same (from looking up vague and non-informative together), but I can't be certain.

Answer 1

Gelman et al (2003) say:

there has long been a desire for prior distributions that can be guaranteed to play a minimal role in the posterior distribution. Such distributions are sometimes called 'reference prior distributions' and the prior density is described as vague, flat, or noninformative.[emphasis from original text]

Based on my reading of the discussion of Jeffrey's prior in Gelman et al (2003, pp 62ff., there is no consensus about the existence of a truly non-informative prior, and that sufficiently vague/flat/diffuse priors are sufficient.

Some of the points that they make:

1. Any prior includes information, including priors that state that no information is known.
   • For example, if we know that we know nothing about the parameter in question, then we know something about it.
2. In most applied context, there is no clear advantage to a truly non-informative prior when sufficiently vague priors suffice, and in many cases there are advantages - like finding a proper prior - to using a vague parameterization of a conjugate prior.
3. Jeffrey's principle can be useful to construct priors that minimize Fisher's information content in univariate models, but there is no analogue for the multivariate case
4. When comparing models, the Jeffrey's prior will vary with the distribution of the likelihood, so priors would also have to change
5. there has generally been a lot of debate about whether a non-informative prior even exists (because of 1, but also see discussion and references on p 66 in Gelman et al for the history of this debate).

note this is community wiki - The underlying theory is at the limits of my understanding, and I would appreciate contributions to this answer.

Gelman et al 2003 Bayesian Data Analysis, Chapman and Hall/CRC

Answer 2

Definitely not, although they are frequently used interchangeably. A vague prior (relatively uninformed, not really favoring some values over others) on a parameter $\theta$ can actually induce a very informative prior on some other transformation $f(\theta)$. This is at least part of the motivation for Jeffrey's prior, which was initially constructed to be as non-informative as possible.

Vague priors can also do some pretty miserable things to your model. The now-classic example is using $InverseGamma(\epsilon, \epsilon)$ as $\epsilon\rightarrow 0$ priors on variance components in a hierarchical model.

The improper limiting prior gives an improper posterior in this case. A popular alternative was to take $\epsilon$ to be really small, which results in a prior that looks almost uniform on $\mathbb{R}^+$. But it also results in a posterior that is almost improper, and model fitting and inferences suffered. See Gelman's Prior distributions for variance parameters in hierarchical models for a complete exposition.

Edit: @csgillespie (rightly!) points out that I haven't completely answered your question. To my mind a non-informative prior is one that is vague in the sense that it doesn't particularly favor one area of the parameter space over another, but in doing so it shouldn't induce informative priors on other parameters. So a non-informative prior is vague but a vague prior isn't necessarily noninformative. One example where this comes into play is Bayesian variable selection; a "vague" prior on variable inclusion probabilities can actually induce a pretty informative prior on the total number of variables included in the model!

It seems to me that the search for truly noninformative priors is quixotic (though many would disagree); better to use so-called "weakly" informative priors (which, I suppose, are generally vague in some sense). Really, how often do we know nothing about the parameter in question?

Answer 3

Lambert et al (2005) raise the question "How Vague is Vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS". They write: "We do not advocate the use of the term non-informative prior distribution as we consider all priors to contribute some information". I tend to agree but I am definitely no expert in Bayesian statistics.

Answer 4

I suspect "vague prior" is used to mean a prior that is known to encode some small, but non-zero amount of knowledge regarding the true value of a parameter, whereas a "non-informative prior" would be used to mean complete ignorance regarding the value of that parameter. It would perhaps be used to show that the analysis was not completely objective.

For example a very broad Gaussian might be a vague prior for a parameter where a non-informative prior would be uniform. The Gaussian would be very nearly flat on the scale of interest, but would nevertheless favour one particular value a bit more than any other (but it might make the problem more mathematically tractable).