27
$\begingroup$

This is a question about terminology. Is a "vague prior" the same as a non-informative prior, or is there some difference between the two? My impression is that they are same (from looking up vague and non-informative together), but I can't be certain.

$\endgroup$
  • 1
    $\begingroup$ Vaguely: Yes. Non-informatively: No. $\endgroup$ – cardinal Nov 1 '13 at 14:41
17
$\begingroup$

Gelman et al. (2003) say:

there has long been a desire for prior distributions that can be guaranteed to play a minimal role in the posterior distribution. Such distributions are sometimes called 'reference prior distributions' and the prior density is described as vague, flat, or noninformative.[emphasis from original text]

Based on my reading of the discussion of Jeffreys' prior in Gelman et al. (2003, p.62ff, there is no consensus about the existence of a truly non-informative prior, and that sufficiently vague/flat/diffuse priors are sufficient.

Some of the points that they make:

  1. Any prior includes information, including priors that state that no information is known.
    • For example, if we know that we know nothing about the parameter in question, then we know something about it.
  2. In most applied contexts, there is no clear advantage to a truly non-informative prior when sufficiently vague priors suffice, and in many cases there are advantages - like finding a proper prior - to using a vague parameterization of a conjugate prior.
  3. Jeffreys' principle can be useful to construct priors that minimize Fisher's information content in univariate models, but there is no analogue for the multivariate case
  4. When comparing models, the Jeffreys' prior will vary with the distribution of the likelihood, so priors would also have to change
  5. there has generally been a lot of debate about whether a non-informative prior even exists (because of 1, but also see discussion and references on p.66 in Gelman et al. for the history of this debate).

note this is community wiki - The underlying theory is at the limits of my understanding, and I would appreciate contributions to this answer.

Gelman et al. 2003 Bayesian Data Analysis, Chapman and Hall/CRC

$\endgroup$
  • $\begingroup$ Gelman himself has moved beyond "noninformative" priors, see eg stat.columbia.edu/~cook/movabletype/archives/2007/07/… and the paper I reference in my answer, and other works of his. This attitude seems increasingly prevalent among applied statisticians. $\endgroup$ – JMS Feb 22 '11 at 16:38
  • $\begingroup$ @JMS Gelman et. al. 2003 made this point, and I have tried to summarize it, they seem to consider the noninformative priors for background / theoretical / heuristic purposes $\endgroup$ – David LeBauer Feb 22 '11 at 16:43
  • $\begingroup$ Yeah, a textbook on Bayesian statistics certainly can't ignore them. I only meant to add that the trend in applied statistics since then has been to move away from these traditional choices, although work continues on "objective Bayes" $\endgroup$ – JMS Feb 22 '11 at 16:57
  • $\begingroup$ I think people tend to use information in two ways when describing priors, in an informal sense and in an information theoretical sense. So you could device a maximum entropy prior that contain as little information as possible, in an information theoretical sense. Using this distribution you (might) signal that "you know nothing about the parameter", this could be quite informative in a more informal sense... $\endgroup$ – Rasmus Bååth Jan 17 '14 at 15:12
9
$\begingroup$

Definitely not, although they are frequently used interchangeably. A vague prior (relatively uninformed, not really favoring some values over others) on a parameter $\theta$ can actually induce a very informative prior on some other transformation $f(\theta)$. This is at least part of the motivation for Jeffreys' prior, which was initially constructed to be as non-informative as possible.

Vague priors can also do some pretty miserable things to your model. The now-classic example is using $\mathrm{InverseGamma}(\epsilon, \epsilon)$ as $\epsilon\rightarrow 0$ priors on variance components in a hierarchical model.

The improper limiting prior gives an improper posterior in this case. A popular alternative was to take $\epsilon$ to be really small, which results in a prior that looks almost uniform on $\mathbb{R}^+$. But it also results in a posterior that is almost improper, and model fitting and inferences suffered. See Gelman's Prior distributions for variance parameters in hierarchical models for a complete exposition.

Edit: @csgillespie (rightly!) points out that I haven't completely answered your question. To my mind a non-informative prior is one that is vague in the sense that it doesn't particularly favor one area of the parameter space over another, but in doing so it shouldn't induce informative priors on other parameters. So a non-informative prior is vague but a vague prior isn't necessarily noninformative. One example where this comes into play is Bayesian variable selection; a "vague" prior on variable inclusion probabilities can actually induce a pretty informative prior on the total number of variables included in the model!

It seems to me that the search for truly noninformative priors is quixotic (though many would disagree); better to use so-called "weakly" informative priors (which, I suppose, are generally vague in some sense). Really, how often do we know nothing about the parameter in question?

$\endgroup$
  • $\begingroup$ Your answer is very nice, but you don't actually state what a "non-informative prior" and how it differs from a "vague prior";) $\endgroup$ – csgillespie Feb 22 '11 at 16:16
  • $\begingroup$ @cgillespie: You're right, edited :) $\endgroup$ – JMS Feb 22 '11 at 16:39
  • $\begingroup$ following from your last paragraph and supporting the quixotic nature of the search: if we know that we know nothing about the parameter in question, then we know something about it. $\endgroup$ – David LeBauer Feb 22 '11 at 16:46
  • $\begingroup$ Quite! The great irony is that many who spend so much energy hand-wringing over their choice of prior don't think twice before throwing down the likelihood. A misspecified likelihood is much more liable to cause troubles, generally. But that's another question entirely... $\endgroup$ – JMS Feb 22 '11 at 16:53
5
$\begingroup$

Lambert et al (2005) raise the question "How Vague is Vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS". They write: "We do not advocate the use of the term non-informative prior distribution as we consider all priors to contribute some information". I tend to agree but I am definitely no expert in Bayesian statistics.

$\endgroup$
  • 2
    $\begingroup$ That agrees with my subjective prior belief that there's no such thing as a non-informative prior. $\endgroup$ – onestop Feb 22 '11 at 15:58
  • $\begingroup$ @onestop please consider contributing to my CW answer $\endgroup$ – David LeBauer Feb 22 '11 at 16:17
  • $\begingroup$ This is precisely the issue addressed in the paper I included in my answer ($IG(\epsilon, \epsilon)$ with $\epsilon \rightarrow 0$. The WinBUGS documentation used them long after these priors were discredited. $\endgroup$ – JMS Feb 22 '11 at 16:34
3
$\begingroup$

I suspect "vague prior" is used to mean a prior that is known to encode some small, but non-zero amount of knowledge regarding the true value of a parameter, whereas a "non-informative prior" would be used to mean complete ignorance regarding the value of that parameter. It would perhaps be used to show that the analysis was not completely objective.

For example a very broad Gaussian might be a vague prior for a parameter where a non-informative prior would be uniform. The Gaussian would be very nearly flat on the scale of interest, but would nevertheless favour one particular value a bit more than any other (but it might make the problem more mathematically tractable).

$\endgroup$
  • $\begingroup$ this begs the question of would the uniform be truly non-informative? what would be the max and min of the non-informative uniform? Would the prior be non-informative after transformation to another scale? $\endgroup$ – David LeBauer Feb 22 '11 at 16:16
  • $\begingroup$ Uniform priors are not necesarily un-informative and vice versa, it depends on the nature of the problem. I think the idea of a minimally informative prior is often more realistic, using MAXENT and transformation groups to decide on a prior distribution that conveys the minimum amount of information consisten with known constraints (e.g. that the prior should be invariant to scaling). For many problems, improper priors work well, so there isn't necessarily a need for there to be a well defined minimum and maximum. $\endgroup$ – Dikran Marsupial Feb 22 '11 at 16:37
2
$\begingroup$

Non-informative priors have different forms. These forms include vague priors and improper priors. So vague prior is part of non-informative priors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.