Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 7 down vote favorite
4

Consider a Jeffreys prior where $p(\theta) \propto \sqrt{|i(\theta)|}$, where $i$ is the Fisher information.

I keep seeing this prior being mentioned as a uninformative prior, but I never saw an argument why it is uninformative. After all, it is not a constant prior, so there has to be some other argument.

I understand that it does not depends on reparametrization, which brings me to the next question. Is it that the determinant of the Fisher information does not depend on reparametrization? Because Fisher information definitely depends on the parametrization of the problem.

Thanks.

share|improve this question
Have you read the Wikipedia article? en.wikipedia.org/wiki/Jeffreys_prior – whuber♦ Feb 22 '11 at 23:12
1  
Yes, I had looked there. perhaps I am missing something, but I do not feel that the Wikipedia article gives an adequate answer to my questions. – bayesian Feb 22 '11 at 23:23
Note that the Jeffreys prior is not invariant with respect to equivalent models. For example Inference about a parameter $p$ is different when using binomial or negative binomial sampling distributions. This is despite the likelihood functions being proportional and the parameter having the same meaning in both models. – probabilityislogic Oct 11 '12 at 6:41

3 Answers

up vote 3 down vote accepted

It's considered noninformative because of the parameterization invariance. You seem to have the impression that a uniform (constant) prior is noninformative. Sometimes it is, sometimes it isn't.

What happens with Jeffrey's prior under a transformation is that the Jacobian from the transformation gets sucked into the original Fisher information, which ends up giving you the Fisher information under the new parameterization. No magic (in the mechanics at least), just a little calculus and linear algebra.

share|improve this answer
1  
I disagree with this answer. Using a subjective prior is also a parametrization invariant procedure ! – Stéphane Laurent Oct 10 '12 at 11:26
up vote 8 down vote

The Jeffreys prior coincides with the Bernardo reference prior for one-dimensional parameter space (and "regular" models). Roughly speaking, this is the prior for which the Kullback-Leibler divergence between the prior and the posterior is maximal. This quantity represents the amount of information brought by the data. This is why the prior is considered to be uninformative: this is the one for which the data brings the maximal amount of information.

By the way I don't know whether Jeffreys was aware of this characterization of his prior ?

share|improve this answer
" Roughly speaking, this is the prior for which the Kullback-Leibler divergence between the prior and the posterior is maximal." Interesting, I did not know that. – Cam.Davidson.Pilon Oct 10 '12 at 14:57
(+1) Good answer. It would be nice to see some references of some of your points (e.g. 1, 2). – user10525 Oct 10 '12 at 18:10
1  
@Procrastinator I am currently writing a new post about noninformative priors ;) Please wait, perhaps a few days. – Stéphane Laurent Oct 10 '12 at 18:14
up vote 1 down vote

I'd say it isn't absolutely non-informative, but minimally informative. It encodes the (rather weak) prior knowledge that you know your prior state of knowledge doesn't depend on its parameterisation (e.g. the units of measurement). If your prior state of knowledge was precisely zero, you wouldn't know that your prior was invariant to such transformations.

share|improve this answer
I am confused. In what sort of case would you know that you prior should depend on the model parameterization? – John Lawrence Aspden Apr 30 at 19:42
If we want to predict longevity as a function of body weight, using a GLM, we know that the conclusion should not be affected whether we weigh the subject in kg or lb; if you use a simple uniform prior over the weights you might get different outcome depending on the units of measurement. – Dikran Marsupial May 1 at 10:08
That's a case when you know that it shouldn't be affected. What is a case where it should? – John Lawrence Aspden May 3 at 17:21
I think you are missing my point. Say we don't know anything about the attributes, not even that they have units of measurement to which the analysis should be invariant. In that case your prior would encode less information about the problem than the Jeffrey's prior, hence the Jeffrey's prior is not completely uninformative. The may or may not be situations where the analysis should not be invariant to some transformation, but that is beside the point. – Dikran Marsupial May 3 at 17:51
N.B according to the BUGS book (p83), Jeffrey's himself referred to such transformation invariant priors as being "minimally informative", which implies that he saw them as encoding some information about the problem. – Dikran Marsupial May 3 at 18:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.