24
$\begingroup$

I am writing a short theoretical essay for a Bayesian Statistics course (in an Economics M.Sc.) on uninformative priors and I am trying to understand which are the steps in the development of this theory.

By now, my timeline is made three main steps: Laplace's indifference principle (1812), Non-Invariant priors (Jeffreys (1946)), Bernardo reference prior (1979).

From my literature review, I've understood that indifference principle (Laplace) was the first tool used to represent lack of prior information but the missing requirement of invariance has led to its abandonment until the 40s, when Jeffreys introduced his method, which has the desired property of invariance. The arise of paradoxes of marginalization due to the careless use of improper prior in the 70s pushed Bernardo to elaborate his reference prior theory to deal with this issue.

Reading the literature, every author cites different contributes: Jaynes' maximum entropy, Box and Tiao's data-translated likelihood, Zellner, ...

In your opinion, what are the crucial steps I am missing?

EDIT: I add my (main) references, if someone needs:

1) The selection of prior by formal rules, Kass, Wasserman

2) A catalogue of non informative priors, Yang, Berger

3) Noninformative Bayesians Priors Interpretation and Problems with Construction and Applications

$\endgroup$
  • 10
    $\begingroup$ once you have finished that theoretical essay, would you be some kind as to link it here? $\endgroup$ – Nikolas Rieble Nov 18 '16 at 16:07
  • 2
    $\begingroup$ It would be great if you could provide answer for your own question summarizing your thesis. $\endgroup$ – Tim Nov 18 '16 at 18:47
  • 3
    $\begingroup$ I've linked this article before, but the epic history of maximum likelihood, covers the historical "gap" between Laplace and Jeffrey's: where the work of Gauss, Hotelling, Fisher, Bernoulli, and others pointed estimation toward maximum likelihood during that time. $\endgroup$ – AdamO Nov 18 '16 at 20:12
  • 2
    $\begingroup$ @alessandro it describes how the Laplacian approach was maintained for basically a century after Gauss developed and used uniform prints (conceiving them as noninformative). Pearson and Kristine Smith disavowed ML because the resulting inference did not deal with probabilities as a Bayesian would desire. $\endgroup$ – AdamO Nov 19 '16 at 20:57
  • 7
    $\begingroup$ Minute (pedantic, if you like) but possibly useful point: Jeffreys = (Professor Sir) Harold Jeffreys, British applied mathematician, geophysicist and much else; he explained to me in a letter 40 years ago that he preferred the possessive Jeffreys's because Jeffreys' was liable to mutation to the quite incorrect Jeffrey's. Above we have an example! (It doesn't help that Richard C. Jeffrey, American philosopher, an entirely different person, also wrote on probability.) $\endgroup$ – Nick Cox Nov 29 '16 at 19:11
13
+100
$\begingroup$

What you seem to be missing is the early history. You can check the paper by Fienberg (2006) When Did Bayesian Inference Become "Bayesian"?. First, he notices that Thomas Bayes was the first one who suggested using a uniform prior:

In current statistical language, Bayes' paper introduces a uniform prior distribution on the binomial parameter, $\theta$, reasoning by analogy with a "billiard table" and drawing on the form of the marginal distribution of the binomial random variable, and not on the principle of "insufficient reason," as many others have claimed.

Pierre Simon Laplace was the next person to discuss it:

Laplace also articulated, more clearly than Bayes, his argument for the choice of a uniform prior distribution, arguing that the posterior distribution of the parameter $\theta$ should be proportional to what we now call the likelihood of the data, i.e.,

$$ f(\theta\mid x_1,x_2,\dots,x_n) \propto f(x_1,x_2,\dots,x_n\mid\theta) $$

We now understand that this implies that the prior distribution for $\theta$ is uniform, although in general, of course, the prior may not exist.

Moreover Carl Friedrich Gauss also referred to using an uninformative prior, as noted by David and Edwards (2001) in their book Annotated Readings in the History of Statistics:

Gauss uses an ad hoc Bayesian-type argument to show that the posterior density of $h$ is proportional to the likelihood (in modern terminology):

$$ f(h|x) \propto f(x|h) $$

where he has assumed $h$ to be uniformly distributed over $[0, \infty)$. Gauss mentions neither Bayes nor Laplace, although the latter had popularized this approach since Laplace (1774).

and as Fienberg (2006) notices, "inverse probability" (and what follows, using uniform priors) was popular at the turn of the 19th century

[...] Thus, in retrospect, it shouldn't be surprising to see inverse probability as the method of choice of the great English statisticians of the turn of the century, such as Edgeworth and Pearson. For example, Edgeworth (49) gave one of the earliest derivations of what we now know as Student's $t$-distribution, the posterior distribution of the mean $\mu$ of a normal distribution given uniform prior distributions on $\mu$ and $h =\sigma^{-1}$ [...]

The early history of the Bayesian approach is also reviewed by Stigler (1986) in his book The history of statistics: The measurement of uncertainty before 1900.

In your short review you also do not seem to mention Ronald Aylmer Fisher (again quoted after Fienberg, 2006):

Fisher moved away from the inverse methods and towards his own approach to inference he called the "likelihood," a concept he claimed was distinct from probability. But Fisher's progression in this regard was slow. Stigler (164) has pointed out that, in an unpublished manuscript dating from 1916, Fisher didn't distinguish between likelihood and inverse probability with a flat prior, even though when he later made the distinction he claimed to have understood it at this time.

Jaynes (1986) provided his own short review paper Bayesian Methods: General Background. An Introductory Tutorial that you could check, but it does not focus on uninformative priors. Moreover, as noted by AdamO, you should definitely read The Epic Story of Maximum Likelihood by Stigler (2007).

It is also worth mentioning that there is no such thing as an "uninformative prior", so many authors prefer talking about "vague priors", or "weekly informative priors".

A theoretical review is provided by Kass and Wasserman (1996) in The selection of prior distributions by formal rules, who go into greater detail about choosing priors, with extended discussion of usage of uninformative priors.

$\endgroup$
  • $\begingroup$ That was the kind of answer I was looking for. Thank you! $\endgroup$ – PhDing Dec 3 '16 at 15:31
  • $\begingroup$ I think Fienberg extended the proud of Bayesians too far. I personally strongly dislike using "inverse probability" to define anything because it does not seem to be consistent with the integral geometry picture proposed by Adler and Taylor. Any good statistical procedure should have its mathematical correspondence, inverse probability is so twisted that you can hardly analyze it when the problem is slightly more sensitive by my experience. $\endgroup$ – Henry.L Dec 5 '16 at 23:26
  • $\begingroup$ @Henry.L ...nevertheless, it is a part of history of statistical thought :) Notice also that it's not only Fienberg who provides such examples. The whole anti-inverse-probability and anti-Bayesian rebel started because it became quite popular. $\endgroup$ – Tim Dec 6 '16 at 13:53
  • $\begingroup$ @Tim Yeah, I guess that is what Thomas Kuhn called "shifting of scheme" and also known as "...opponents eventually die, and a new generation grows up" :)). $\endgroup$ – Henry.L Dec 7 '16 at 2:11
5
$\begingroup$

A few comments about flaws of noninformative priors (uninformative priors) are probably a good idea since the investigation of such flaws helped development of the concept of noninformative prior in history.

You may want to add some comments about the drawbacks/flaws of adopting noninformative priors. Among many criticisms I point out two.

(1) Generally the adoption of noninformative priors has consistency problems especially when the model distribution has multi-modal behavior.

This problem is not unique to noninformative priors but is shared by many other Bayesian procedures as pointed out in the following paper along with its discussions.

Diaconis, Persi, and David Freedman. "On the consistency of Bayes estimates." The Annals of Statistics (1986): 1-26.

Nowadays the noninformative prior is no longer a research focus. It seems that there is more interest in more flexible choices of prior in nonparametric settings. Examples are the Gaussian process prior in nonparametric Bayes procedure or a flexible model like a mixture of Dirichlet priors, as in

Antoniak, Charles E. "Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems." The annals of statistics (1974): 1152-1174.

But again such a prior has its own consistency problems.

(2) Most so-called "noninformative priors" are not well-defined.

This is probably the most evident problem associated with noninformative priors during their development.

One example is that the limit definition of noninformative prior as a limit of a sequence of proper priors will lead to a marginalization paradox. As you mentioned, Bernardo's reference prior also has the problem that Berger never proved that its formal definition is independent from its construction/partition. See the discussion in

Berger, James O., José M. Bernardo, and Dongchu Sun. "The formal definition of reference priors." The Annals of Statistics (2009): 905-938.

One best definition about Jeffreys' prior that is well-defined is that it is chosen to be a prior such that it is invariant under certain parallel translation over the Riemannian manifold equipped with Fisher information metric, but even that does not solve the first problem.

Also you may want to read my explanation about marginalization paradox.

$\endgroup$
  • $\begingroup$ This is an excellent post and none of us thought about it. Great job. $\endgroup$ – Dave Harris Dec 5 '16 at 14:36
  • $\begingroup$ I've made several small edits to expression without trying to change any meaning or implication. Please check that your meaning is invariant under editing. $\endgroup$ – Nick Cox Dec 6 '16 at 18:45
4
$\begingroup$

I would have posted in the comments, but I guess I do not have the reputation yet. The only missing thing, not in the comments already marked, is a special case of noninformative priors whose origins that I have tried to hunt down and have not found. It may precede Jeffreys paper.

For the normal distribution, I have seen the Cauchy distribution used as a noninformative prior for data with a normal likelihood. The reason is that the precision of the Cauchy distribution is zero, where precision is one divided by the variance. It creates a rather peculiar set of contradictory concepts.

The formula for the Cauchy is $$\frac{1}{\pi}\frac{\Gamma}{\Gamma^2+(x-\mu)^2}.$$

Depending on how you define the integral there is either no defined variance or it goes to infinity about the median, which implies the precision goes to zero. In conjugate updating, which wouldn't apply here, you add the weighted precisions. I think this is why this idea of a proper prior with a perfectly imprecise density formed. It is also equivalent to Student's t with one degree of freedom, which could also be the source.

This is a strange idea in the sense that the Cauchy distribution has a well defined center of location and inter-quartile range, which is $2\Gamma$.

The two earliest references to the Cauchy distribution are as likelihood functions. The first in a letter from Poisson to Laplace as an exception to the Central Limit Theorem. The second was in 1851 journal articles in a battle between Bienayme' and Cauchy over the validity of ordinary least squares.

I have found references to its use as a noninformative prior back into the 1980's but I cannot find a first article or book. I also have not found a proof that it is noninformative. I did find a citation to Jeffreys' 1961 book on probability theory, but I have never requested the book via interlibrary loan.

It may be simply weakly informative. The 99.99% highest density region is 1272 semi-interquartile ranges wide.

I hope it helps. It is a weird special case, but you see it come up in a number of regression papers. It satisfies the requirements for a Bayes action by being a proper prior, while minimally influencing location and scale.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.