# What is an “uninformative prior”? Can we ever have one with truly no information?

Inspired by a comment from this question:

What do we consider "uninformative" in a prior - and what information is still contained in a supposedly uninformative prior?

I generally see the prior in an analysis where it's either a frequentist-type analysis trying to borrow some nice parts from Bayesian analysis (be it some easier interpretation all the way to 'its the hot thing to do'), the specified prior is a uniform distribution across the bounds of the effect measure, centered on 0. But even that asserts a shape to the prior - it just happens to be flat.

Is there a better uninformative prior to use?

-
Maybe you’ll enjoy a look on the so-called Principle of Maximum Entropy. I don’t feel like expanding that in a full answer – the Wikipedia article seems of good quality. I’m pretty confident some contributors will expand on it that much better than I would. – Elvis Jan 3 '12 at 9:54

[Warning: as a card-carrying member of the Objective Bayes Section of ISBA, my views are not representative of all Bayesian statisticians!, quite the opposite...]

In summary, there is no such thing as a prior with "truly no information".

Indeed, the "uninformative" prior is sadly a misnomer. Any prior distribution contains some specification that is akin to some amount of information. Even (or especially) the uniform prior. Indeed, the uniform prior is only flat for one given parameterisation of the problem. If one changes to another parameterisation (even a bounded one), the Jacobian change of variable comes into the picture and density and the prior is flat no longer.

As pointed out by Elvis, maximum entropy is one approach advocated to select so-called "uninformative" priors. It however requires (a) enough information on some moments $h(\theta)$ of the prior distribution $\pi(\cdot)$ to specify the constraints$$\int_{\Theta} h(\theta)\,\text{d}\pi(\theta) = \mathfrak{h}_0$$ that lead to the MaxEnt prior $$\pi^*(\theta)\propto \exp\{ \lambda^\text{T}h(\theta) \}$$ and (b) the preliminary choice of a reference measure $\text{d}\mu(\theta)$ [in continuous settings], a choice that brings the debate back to its initial stage! (In addition, the parametrisation of the constraints (i.e., the choice of $h$) impacts the shape of the resulting MaxEnt prior.)

José Bernardo has produced an original theory of reference priors where he chooses the prior in order to maximise the information brought by the data by maximising the Kullback distance between prior and posterior. In the simplest cases with no nuisance parameters, the solution is Jeffreys' prior. In more complex problems, (a) a choice of the parameters of interest (or even a ranking of their order of interest) must be made; (b) the computation of the prior is fairly involved and requires a sequence of embedded compact sets to avoid improperness issues. (See e.g. The Bayesian Choice for details.)

In short, there is no "best" (or even "better") choice for "the" "uninformative" prior. And I consider this is how things should be because the very nature of Bayesian analysis implies that the choice of the prior distribution matters. And that there is no comparison of priors: one cannot be "better" than another. (At least before observing the data: once it is observed, comparison of priors becomes model choice.) The conclusion of José Bernardo, Jim Berger, Dongchu Sun, and many other "objective" Bayesians is that there are roughly equivalent reference priors one can use when being unsure about one's prior information or seeking a benchmark Bayesian inference, some of those priors being partly supported by information theory arguments, others by non-Bayesian frequentist properties (like matching priors), and all resulting in rather similar inferences.

-
(+1) Your book? Oh damn. I so have 387 questions for you :) – Elvis Jan 3 '12 at 13:07
(+1) For an objective (no less!), straightforward answer. – cardinal Jan 3 '12 at 15:00
+1 Thank you for a good and well-informed overview of the issues. – whuber Jan 3 '12 at 15:00
An outstanding answer. Thank you. And yet another book to go on the wish list. – Fomite Jan 3 '12 at 18:26
It's almost unfair. After all, he's Christian Robert! Just kidding. Great answer. And I'd love if @Xi'an could expand it in a post at his blog, specially about how parametrization is important to the topic of "uninformative" priors. – Manoel Galdino Jan 13 '12 at 19:35

An appealing property of formal noninformative priors is the "frequentist-matching property" : it means that a posterior 95%-credibility interval is also (at least, approximately) a 95%-confidence interval in the frequentist sense. This property holds for Bernardo's reference prior although the fundations of these noninformative priors are not oriented towards the achievement of a good frequentist-matching property, If you use a "naive" ("flat") noninformative prior such as the uniform distribution or a Gaussian distribution with a huge variance then there is no guarantee that the frequentist-matching property holds. Maybe Bernardo's reference prior could not be considered as the "best" choice of a noninformative prior but could be considered as the most successful one. Theoretically it overcomes many paradoxes of other candidates.

-

Jeffreys distributions also suffer from inconsistencies: the Jeffreys priors for a variable over $(-\infty,\infty)$ or over $(0,\infty)$ are improper, which is not the case for the Jeffreys prior of a probability parameter $p$: the measure $\text{d}p/\sqrt{p(1-p)}$ has a mass of $\pi$ over $(0,1)$.

Renyi has shown that a non-informative distribution must be associated with an improper integral. See instead Lhoste's distributions which avoid this difficulty and are invariant under changes of variables (e.g., for $p$, the measure is $\text{d}p/p(1-p)$).

First, the translation is good !

For E. LHOSTE : "Le calcul des probabilités appliqué à l'artillerie", Revue d'artillerie, tome 91, mai à août 1923

For A. RENYI : "On a new axiomatic theory of probability" Acta Mathematica, Académie des Sciences hongroises, tome VI, fasc.3-4, 1955

I can add : M. DUMAS : "Lois de probabilité a priori de Lhoste", Sciences et techniques de l'armement, 56, 4ème fascicule, 1982, pp 687-715

-
Is it possible for you to re-write this in English, even if it is done quite poorly through an automated translation service like Google Translate? Other users, more fluent in both French and English, can help copy-edit it for you. – Silverfish Nov 6 '15 at 19:30
As far as I remember, Lhoste's invariance result is restricted to the transforms $\log\sigma$ and $\log p/(1-p)$ for parameters on $(0,\infty)$ and $(0,1)$, respectively. Other transforms from $(0,\infty)$ and $(0,1)$ to $\mathbb{R}$ will result in different priors. – Xi'an Nov 6 '15 at 21:23
From my brief correspondence with Maurice Dumas in the early 1990's, I remember that he wrote a Note aux Comptes-Rendus de l'Académie des Sciences, where he uses the $\log()$ and $\text{logit}()$ transforms to derive "invariant" priors. – Xi'an Nov 9 '15 at 18:53