What is an “uninformative prior”? Can we ever have one with truly no information?

Question

Inspired by a comment from this question:

What do we consider "uninformative" in a prior - and what information is still contained in a supposedly uninformative prior?

I generally see the prior in an analysis where it's either a frequentist-type analysis trying to borrow some nice parts from Bayesian analysis (be it some easier interpretation all the way to 'its the hot thing to do'), the specified prior is a uniform distribution across the bounds of the effect measure, centered on 0. But even that asserts a shape to the prior - it just happens to be flat.

Is there a better uninformative prior to use?

Answer 1

The "uninformative" prior is sadly a misnomer. Any prior distribution contains some specification that is akin to some amount of information. Even the uniform prior. Indeed, the uniform prior is only flat for one parameterisation of the problem. If one changes to another parameterisation (even a bounded one), the Jacobian comes into play and the prior is flat no longer.

As pointed out by Elvis, maximum entropy is one approach to select so-called "uninformative" priors. It however requires (a) some level of information on some moments of the prior to specify the condtraints and (b) the choice of a reference measure in continuous settings, which brings the debate back to its initial stage!

José Bernardo has produced an original theory of reference priors where he chooses the prior in order to maximise the information brought by the data. In the simplest cases, the solution is Jeffreys' prior. In more complex problems, (a) a choice of the parameters of interest must be made; (b) the computation of the prior is fairly involved. (See e.g. my book The Bayesian Choice for details.)

In short, there is no "best" (or even "better") choice for "the" "uninformative" prior. And I think there should not be because the very nature of Bayesian analysis implies that the choice of the prior distribution matters. And that there is no comparison of priors: one cannot be "better" than another. The conclusion of José Bernardo, Jim Berger and many other "objective" Bayesians is that there are roughly equivalent reference priors one can use when unsure about one's prior information, some being backed up by information theory arguments, others by non-Bayesian frequentist properties, and conducting to rather similar inferences.

Answer 2

An appealing property of formal noninformative priors is the "frequentist-matching property" : it means that a posterior 95%-credibility interval is also (at least, approximately) a 95%-confidence interval in the frequentist sense. This property holds for Bernardo's reference prior although the fundations of these noninformative priors are not oriented towards the achievement of a good frequentist-matching property, If you use a "naive" ("flat") noninformative prior such as the uniform distribution or a Gaussian distribution with a huge variance then there is no guarantee that the frequentist-matching property holds. Maybe Bernardo's reference prior could not be considered as the "best" choice of a noninformative prior but could be considered as the most successful one. Theoretically it overcomes many paradoxes of other candidates.