What is the point of non-informative priors?

Question

Why even have non-informative priors? They don't provide information about $\theta$. So why use them? Why not only use informative priors? For example, suppose $ \theta \in [0,1]$. Then is $\theta \sim \mathcal{U}(0,1)$ a non-informative prior for $\theta$?

Answer 1

The debate about non-informative priors has been going on for ages, at least since the end of the 19th century with criticism by Bertrand and de Morgan about the lack of invariance of Laplace's uniform priors (the same criticism reported by Stéphane Laurent in the above comments). This lack of invariance sounded like a death stroke for the Bayesian approach and, while some Bayesians were desperately trying to cling to specific distributions, using less than formal arguments, others had a vision of a larger picture where priors could be used in situations where there was hardly any prior information, besides the shape of the likelihood itself.

This vision is best represented by Jeffreys' distributions, where the information matrix of the sampling model, $I(\theta)$, is turned into a prior distribution $$ \pi(\theta) \propto |I(\theta)|^{1/2} $$ which is most often improper, i.e. does not integrate to a finite value. The label "non-informative" associated with Jeffreys' priors is rather unfortunate, as they represent an input from the statistician, hence are informative about something! Similarly, "objective" has an authoritative weight I dislike... I thus prefer the label "reference prior", used for instance by José Bernado.

Those priors indeed give a reference against which one can compute either the reference estimator/test/prediction or one's own estimator/test/prediction using a different prior motivated by subjective and objective items of information. To answer directly the question, "why not use only informative priors?", there is actually no answer. A prior distribution is a choice made by the statistician, neither a state of Nature nor a hidden variable. In other words, there is no "best prior" that one "should use". Because this is the nature of statistical inference that there is no "best answer".

Hence my defence of the noninformative/reference choice! It is providing the same range of inferential tools as other priors, but gives answers that are only inspired by the shape of the likelihood function, rather than induced by some opinion about the range of the unknown parameters.

Answer 2

I'll try to give some background for Xi'an's answer.

In the abstract of Prior Probabilities (1968), E. T. Jaynes says,

In decision theory, mathematical analysis shows that once the sampling distributions, loss function, and sample are specified, the only remaining basis for a choice among different admissible decisions lies in the prior probabilities. Therefore, the logical foundations of decision theory cannot be put in fully satisfactory form until the old problem of arbitrariness (sometimes called "subjectiveness") in assigning prior probabilities is resolved.

Objective Bayesians like Jaynes see statistical inference as an extension of logic, so they believe that two competent statisticians, faced with the same problem and the same information, should arrive at the same conclusions. But on the standard account of Bayesian statistical inference, there's no principled way to choose a prior in the absence of any information at all.

So Jaynes wanted to find a collection of principles that would be enough to uniquely determine an appropriate prior probability distribution for a given statistical inference problem in the absence of any data. In the late 18th century, Laplace had proposed the "principle of insufficient reason": you should assign equal probabilities to events which are distinguished only by their labels. Jaynes generalized and formalized that principle, and he also argued for another, the "principle of maximum entropy". The idea of both of these principles is that you should assign prior probabilities in a way that introduces the least possible additional information to the problem.

Here's a toy example related to your question about whether the uniform distribution is non-informative for a parameter constrained to the interval [0, 1]: Suppose you're presented with a coin of unknown bias θ, and you don't know anything else about it. You haven't observed any flips yet. What should you believe about the value of θ? Well, you could always just call heads "tails" and vice versa, so by Laplace's principle we know the prior distribution on θ must be symmetric about 1/2. Jaynes gets us further: it's not hard to prove that the uniform distribution is the one that maximizes entropy for a continuous parameter constrained to an interval. So it is indeed non-informative in that specific sense.

Unfortunately for Jaynes, skeptics of the objective Bayesian framework have pointed out a number of issues with it. One well-known one that Stéphane alludes to in his comment is van Fraasen's "cube factory paradox" from Laws and Symmetry (1989):

A precision tool factory produces iron cubes with edge length ≤ 2 cm. What is the probability that a cube has length ≤ 1 cm given that it was produced by that factory?

The problem here is that the uniform prior on edge length is not uniform anymore if we parameterize this problem in terms of area or volume instead, and then translate back to side length. Jefferey's priors are an attempt to resolve this sort of issue, which brings us to the beginning of Xi'an's answer.