69
$\begingroup$

Inspired by a comment from this question:

What do we consider "uninformative" in a prior - and what information is still contained in a supposedly uninformative prior?

I generally see the prior in an analysis where it's either a frequentist-type analysis trying to borrow some nice parts from Bayesian analysis (be it some easier interpretation all the way to 'its the hot thing to do'), the specified prior is a uniform distribution across the bounds of the effect measure, centered on 0. But even that asserts a shape to the prior - it just happens to be flat.

Is there a better uninformative prior to use?

$\endgroup$
  • 2
    $\begingroup$ Maybe you’ll enjoy a look on the so-called Principle of Maximum Entropy. I don’t feel like expanding that in a full answer – the Wikipedia article seems of good quality. I’m pretty confident some contributors will expand on it that much better than I would. $\endgroup$ – Elvis Jan 3 '12 at 9:54
89
+50
$\begingroup$

[Warning: as a card-carrying member of the Objective Bayes Section of ISBA, my views are not representative of all Bayesian statisticians!, quite the opposite...]

In summary, there is no such thing as a prior with "truly no information".

Indeed, the "uninformative" prior is sadly a misnomer. Any prior distribution contains some specification that is akin to some amount of information. Even (or especially) the uniform prior. Indeed, the uniform prior is only flat for one given parameterisation of the problem. If one changes to another parameterisation (even a bounded one), the Jacobian change of variable comes into the picture and density and the prior is flat no longer.

As pointed out by Elvis, maximum entropy is one approach advocated to select so-called "uninformative" priors. It however requires (a) enough information on some moments $h(\theta)$ of the prior distribution $\pi(\cdot)$ to specify the constraints$$\int_{\Theta} h(\theta)\,\text{d}\pi(\theta) = \mathfrak{h}_0$$ that lead to the MaxEnt prior $$\pi^*(\theta)\propto \exp\{ \lambda^\text{T}h(\theta) \}$$ and (b) the preliminary choice of a reference measure $\text{d}\mu(\theta)$ [in continuous settings], a choice that brings the debate back to its initial stage! (In addition, the parametrisation of the constraints (i.e., the choice of $h$) impacts the shape of the resulting MaxEnt prior.)

José Bernardo has produced an original theory of reference priors where he chooses the prior in order to maximise the information brought by the data by maximising the Kullback distance between prior and posterior. In the simplest cases with no nuisance parameters, the solution is Jeffreys' prior. In more complex problems, (a) a choice of the parameters of interest (or even a ranking of their order of interest) must be made; (b) the computation of the prior is fairly involved and requires a sequence of embedded compact sets to avoid improperness issues. (See e.g. The Bayesian Choice for details.)

In an interesting twist, some researchers outside the Bayesian perspective have been developing procedures called confidence distributions that are probability distributions on the parameter space, constructed by inversion from frequency-based procedures without an explicit prior structure or even a dominating measure on this parameter space. They argue that this absence of well-defined prior is a plus, although the result definitely depends on the choice of the initialising frequency-based procedure

In short, there is no "best" (or even "better") choice for "the" "uninformative" prior. And I consider this is how things should be because the very nature of Bayesian analysis implies that the choice of the prior distribution matters. And that there is no comparison of priors: one cannot be "better" than another. (At least before observing the data: once it is observed, comparison of priors becomes model choice.) The conclusion of José Bernardo, Jim Berger, Dongchu Sun, and many other "objective" Bayesians is that there are roughly equivalent reference priors one can use when being unsure about one's prior information or seeking a benchmark Bayesian inference, some of those priors being partly supported by information theory arguments, others by non-Bayesian frequentist properties (like matching priors), and all resulting in rather similar inferences.

$\endgroup$
  • 14
    $\begingroup$ (+1) Your book? Oh damn. I so have 387 questions for you :) $\endgroup$ – Elvis Jan 3 '12 at 13:07
  • 4
    $\begingroup$ (+1) For an objective (no less!), straightforward answer. $\endgroup$ – cardinal Jan 3 '12 at 15:00
  • 2
    $\begingroup$ +1 Thank you for a good and well-informed overview of the issues. $\endgroup$ – whuber Jan 3 '12 at 15:00
  • 2
    $\begingroup$ An outstanding answer. Thank you. And yet another book to go on the wish list. $\endgroup$ – Fomite Jan 3 '12 at 18:26
  • 1
    $\begingroup$ It's almost unfair. After all, he's Christian Robert! Just kidding. Great answer. And I'd love if @Xi'an could expand it in a post at his blog, specially about how parametrization is important to the topic of "uninformative" priors. $\endgroup$ – Manoel Galdino Jan 13 '12 at 19:35
16
$\begingroup$

An appealing property of formal noninformative priors is the "frequentist-matching property" : it means that a posterior 95%-credibility interval is also (at least, approximately) a 95%-confidence interval in the frequentist sense. This property holds for Bernardo's reference prior although the fundations of these noninformative priors are not oriented towards the achievement of a good frequentist-matching property, If you use a "naive" ("flat") noninformative prior such as the uniform distribution or a Gaussian distribution with a huge variance then there is no guarantee that the frequentist-matching property holds. Maybe Bernardo's reference prior could not be considered as the "best" choice of a noninformative prior but could be considered as the most successful one. Theoretically it overcomes many paradoxes of other candidates.

$\endgroup$
8
$\begingroup$

Jeffreys distributions also suffer from inconsistencies: the Jeffreys priors for a variable over $(-\infty,\infty)$ or over $(0,\infty)$ are improper, which is not the case for the Jeffreys prior of a probability parameter $p$: the measure $\text{d}p/\sqrt{p(1-p)}$ has a mass of $\pi$ over $(0,1)$.

Renyi has shown that a non-informative distribution must be associated with an improper integral. See instead Lhoste's distributions which avoid this difficulty and are invariant under changes of variables (e.g., for $p$, the measure is $\text{d}p/p(1-p)$).


First, the translation is good !

For E. LHOSTE : "Le calcul des probabilités appliqué à l'artillerie", Revue d'artillerie, tome 91, mai à août 1923

For A. RENYI : "On a new axiomatic theory of probability" Acta Mathematica, Académie des Sciences hongroises, tome VI, fasc.3-4, 1955

I can add : M. DUMAS : "Lois de probabilité a priori de Lhoste", Sciences et techniques de l'armement, 56, 4ème fascicule, 1982, pp 687-715

$\endgroup$
  • 3
    $\begingroup$ Is it possible for you to re-write this in English, even if it is done quite poorly through an automated translation service like Google Translate? Other users, more fluent in both French and English, can help copy-edit it for you. $\endgroup$ – Silverfish Nov 6 '15 at 19:30
  • 3
    $\begingroup$ As far as I remember, Lhoste's invariance result is restricted to the transforms $\log\sigma$ and $\log p/(1-p)$ for parameters on $(0,\infty)$ and $(0,1)$, respectively. Other transforms from $(0,\infty)$ and $(0,1)$ to $\mathbb{R}$ will result in different priors. $\endgroup$ – Xi'an Nov 6 '15 at 21:23
  • 2
    $\begingroup$ From my brief correspondence with Maurice Dumas in the early 1990's, I remember that he wrote a Note aux Comptes-Rendus de l'Académie des Sciences, where he uses the $\log()$ and $\text{logit}()$ transforms to derive "invariant" priors. $\endgroup$ – Xi'an Nov 9 '15 at 18:53
2
$\begingroup$

I agree with the excellent answer by Xi'an, pointing out that there is no single prior that is "uninformative" in the sense of carrying no information. To expand on this topic, I wanted to point out that one alternative is to undertake Bayesian analysis within the imprecise probability framework (see esp. Walley 1991, Walley 2000). Within this framework the prior belief is represented by a set of probability distributions, and this leads to a corresponding set of posterior distributions. That might sound like it would not be very helpful, but it actually is quite amazing. Even with a very broad set of prior distributions (where certain moments can range over all possible values) you often still get posterior convergence to a single posterior as $n \rightarrow \infty$.

This analytical framework has been axiomatised by Walley as its own special form of probabilistic analysis, but is essentially equivalent to robust Bayesian analysis using a set of priors, yielding a corresponding set of posteriors. In many models it is possible to set an "uninformative" set of priors that allows some moments (e.g., the prior mean) to vary over the entire possible range of values, and this nonetheless produces valuable posterior results, where the posterior moments are bounded more tightly. This form of analysis arguably has a better claim to being called "uninformative", at least with respect to moments that are able to vary over their entire allowable range.


A simple example - Bernoulli model: Suppose we observe data $X_1,...,X_n | \theta \sim \text{IID Bern}(\theta)$ where $\theta$ is the unknown parameter of interest. Usually we would use a beta density as the prior (both the Jeffrey's prior and reference prior are of this form). We can specify this form of prior density in terms of the prior mean $\mu$ and another parameter $\kappa > 1$ as:

$$\begin{equation} \begin{aligned} \pi_0(\theta | \mu, \kappa) = \text{Beta}(\theta | \mu, \kappa) = \text{Beta} \Big( \theta \Big| \alpha = \mu (\kappa - 1), \beta = (1-\mu) (\kappa - 1) \Big). \end{aligned} \end{equation}$$

(This form gives prior moments $\mathbb{E}(\theta) = \mu$ and $\mathbb{V}(\theta) = \mu(1-\mu) / \kappa$.) Now, in an imprecise model we could set the prior to consist of the set of all these prior distributions over all possible expected values, but with the other parameter fixed to control the precision over the range of mean values. For example, we might use the set of priors:

$$\mathscr{P}_0 \equiv \Big\{ \text{Beta}(\mu, \kappa) \Big| 0 \leqslant \mu \leqslant 1 \Big\}. \quad \quad \quad \quad \quad$$

Suppose we observe $s = \sum_{i=1}^n x_i$ positive indicators in the data. Then, using the updating rule for the Bernoulli-beta model, the corresponding posterior set is:

$$\mathscr{P}_\mathbf{x} = \Big\{ \text{Beta}\Big( \tfrac{s + \mu(\kappa-1)}{n + \kappa -1}, n+\kappa \Big) \Big| 0 \leqslant \mu \leqslant 1 \Big\}.$$

The range of possible values for the posterior expectation is:

$$\frac{s}{n + \kappa-1} \leqslant \mathbb{E}(\theta | \mathbb{x}) \leqslant \frac{s + \kappa-1}{n + \kappa-1}.$$

What is important here is that even though we started with a model that was "uninformative" with respect to the expected value of the parameter (the prior expectation ranged over all possible values), we nonetheless end up with posterior inferences that are informative with respect to the posterior expectation of the parameter (they now range over a narrower set of values). As $n \rightarrow \infty$ this range of values is squeezed down to a single point, which is the true value of $\theta$.

$\endgroup$
  • $\begingroup$ +1. Interesting. What is kappa in the last equation? Should it be kappa star? $\endgroup$ – amoeba Mar 4 at 9:59
  • $\begingroup$ I have edited to remove variation in $\kappa$ to give a simpler model. It should be okay now. $\endgroup$ – Ben Mar 4 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.