Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I understand that the Jeffreys prior is invariant under re-parameterization. However, what I don't understand is why this property is desired.

Why wouldn't you want the prior to change under a change of variables?

share|improve this question
1  
Of possible interest: Why are Jeffreys priors considered noninformative?. –  user10525 Oct 9 '12 at 8:03

3 Answers 3

up vote 11 down vote accepted

Let me complete Zen's answer. I don't very like the notion of "representing ignorance". The important thing is not the Jeffreys prior but the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data. The invariance property is naturally required for the two following points. Consider for instance the binomial model with unknown proportion parameter $\theta$ and odds parameter $\psi=\frac{\theta}{1-\theta}$.

1) The Jeffreys posterior on $\theta$ reflects as best as possible the information about $\theta$ brought by the data. There is a one-to-on correspondence between $\theta$ and $\psi$. Then, transforming the Jeffreys posterior on $\theta$ into a posterior on $\psi$ (via the usual change-of-variables formula) should yield a distribution reflecting as best as possible the information about $\psi$. Thus this distribution should be the Jeffreys posterior about $\psi. This is the invariance property.

2) An important point when drawing conclusions of a statistical analysis is scientific communication. Imagine you give the Jeffreys posterior on $\theta$ to a scientific colleague. But he/she is interested in $\psi$ rather than $\theta$. Then this is not a problem with the invariance property: he/she just has to apply the change-of-variables formula.

share|improve this answer
    
Ah this clears things up a little bit. But is there an intuitively good reason why the posterior for the odds parameter should be the same as the posterior for the proportion parameter? That seems rather unnatural to me. –  tskuzzy Oct 9 '12 at 7:15
    
It is not the same ! One is induced by the other by the change-of-variables formula. There is a one-to-one correspondence between the two parameters. Then the posterior distribution on one of these parameters should induce the posterior distribution on the other. –  Stéphane Laurent Oct 9 '12 at 7:21
    
(+1) Stéphane. The OP seems to be still confused when he says "... should be the same...". The two posteriors are not "the same", what happens is that, for example, in Stéphane's example, you have that $P\{1/3\leq\theta\leq 2/3\mid X=x\}=P\{1/2\leq\psi\leq 2\mid X=x\}$; if you don't have this kind of consistency using defaults (computed) priors, then your priors are a little nutty. –  Zen Oct 9 '12 at 18:05
1  
That makes sense, thanks for the clarification! –  tskuzzy Oct 9 '12 at 20:57
2  
This theory is incomplete and depends on parameter ordering, choice of compact region, and the likelihood function. So it doesn't obey the likelihood principle for example. Also, it is difficult to apply to non-independent data. Further, Bernardo's theory is only complete for 1-d parameter problems. It is probably the best method available currently though. A good competitor is the transformation group approach of Jaynes. –  probabilityislogic Oct 10 '12 at 22:39

Suppose that you and a friend are analyzing the same set of data using a normal model. You adopt the usual parameterization of the normal model using the mean and the variance as parameters, but your friend prefers to parameterize the normal model with the coefficient of variation and the precision as parameters (which is perfectly "legal"). If both of you use Jeffreys's priors, your posterior distribution will be your friend's posterior distribution properly transformed from his parameterization to yours. It is in this sense that the Jeffreys's prior is "invariant"

(By the way, "invariant" is a horrible word; what we really mean is that it is "covariant" in the same sense of tensor calculus/differential geometry, but, of course, this term already has a well established probabilistic meaning, so we can't use it.)

Why is this consistency property desired? Because, if the Jeffreys's prior has any chance of representing ignorance about the value of the parameters in an absolute sense (actually, it doesn't, but for other reasons not related to "invariance"), and not ignorance relatively to a particular parameterization of the model, it must be the case that, no matter which parameterizations we arbitrarily choose to start with, our posteriors should "match" after transformation.

Jeffreys himself violated this "invariance" property routinely when constructing his priors.

This paper has some interesting discussions about this and related subjects.

share|improve this answer
    
+1: Good answer. But, why doesn't the Jeffreys' prior represent ignorance about the value of the parameters? –  Neil G Oct 9 '12 at 5:50
1  
Because it is not even a distribution. It is paradoxical to claim that a distribution reflects ignorance. A distribution always reflects information. –  Stéphane Laurent Oct 9 '12 at 6:21
1  
Another reference: projecteuclid.org/… –  Stéphane Laurent Oct 9 '12 at 6:22

To add some quotations to Zen's great answer: According to Jaynes, the Jeffreys prior is an example of the principle of transformation groups, which results from the principle of indifference:

The essence of the principle is just: (1) we recognize that a probability assignment is a means of describing a certain state i knowledge. (2) If the available evidence gives us no reason to consider proposition $A_1$ either more or less likely than $A_2$, then the only honest-way we can describe that state of knowledge is to assign them equal probabilities: $p_1=p_2$. Any other procedure would be inconsistent in the sense that, by a mere interchange of the labels $(1, 2)$ we could then generate a new problem in which our state of knowledge is the same but in which we are assigning different probabilities…

Now, to answer your question: “Why wouldn't you want the prior to change under a change of variables?”

According to Jaynes, the parametrization is another kind of arbitrary label, and one should not be able to “by a mere interchange of the labels generate a new problem in which our state of knowledge is the same but in which we are assigning different probabilities.”

share|improve this answer
    
Jaynes seems somewhat mystic to me. –  Stéphane Laurent Oct 9 '12 at 6:24
    
1  
Xian received a mail praising Jaynes: ceremade.dauphine.fr/~xian/critic.html It's a pity if you don't read French, this mail is both frightening and funny. The writer seems to have gone crazy by thinking too much about Bayesian statistics ;) –  Stéphane Laurent Oct 9 '12 at 6:34
    
Oui, c'est en faite ma langue maternelle! Thanks for the link :) (Although, I think your English is better than my French) –  Neil G Oct 9 '12 at 6:35
    
Vraiment ? Bon amusement alors :) –  Stéphane Laurent Oct 9 '12 at 6:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.