73
$\begingroup$

I understand that the Jeffreys prior is invariant under re-parameterization. However, what I don't understand is why this property is desired.

Why wouldn't you want the prior to change under a change of variables?

$\endgroup$
1

6 Answers 6

39
$\begingroup$

Let me complete Zen's answer. I don't very like the notion of "representing ignorance". The important thing is not the Jeffreys prior but the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data. The invariance property is naturally required for the two following points. Consider for instance the binomial model with unknown proportion parameter $\theta$ and odds parameter $\psi=\frac{\theta}{1-\theta}$.

  1. The Jeffreys posterior on $\theta$ reflects as best as possible the information about $\theta$ brought by the data. There is a one-to-one correspondence between $\theta$ and $\psi$. Then, transforming the Jeffreys posterior on $\theta$ into a posterior on $\psi$ (via the usual change-of-variables formula) should yield a distribution reflecting as best as possible the information about $\psi$. Thus this distribution should be the Jeffreys posterior about $\psi$. This is the invariance property.

  2. An important point when drawing conclusions of a statistical analysis is scientific communication. Imagine you give the Jeffreys posterior on $\theta$ to a scientific colleague. But he/she is interested in $\psi$ rather than $\theta$. Then this is not a problem with the invariance property: he/she just has to apply the change-of-variables formula.

$\endgroup$
8
  • 1
    $\begingroup$ It is not the same ! One is induced by the other by the change-of-variables formula. There is a one-to-one correspondence between the two parameters. Then the posterior distribution on one of these parameters should induce the posterior distribution on the other. $\endgroup$ Oct 9, 2012 at 7:21
  • 2
    $\begingroup$ (+1) Stéphane. The OP seems to be still confused when he says "... should be the same...". The two posteriors are not "the same", what happens is that, for example, in Stéphane's example, you have that $P\{1/3\leq\theta\leq 2/3\mid X=x\}=P\{1/2\leq\psi\leq 2\mid X=x\}$; if you don't have this kind of consistency using defaults (computed) priors, then your priors are a little nutty. $\endgroup$
    – Zen
    Oct 9, 2012 at 18:05
  • 1
    $\begingroup$ I think what is missing from this post is that when there is a lot of information in the data about a parameter, the particular prior used doesn't really matter. For example, a binomial proportion, whether we use a uniform, jeffreys or haldane prior makes very little difference unless the posterior is very broad. In this case its a bit of an academic argument as to which prior is "right" because no meaningful conclusions can be drawn anyway. The real value of a non-informative prior is in multiple dimensions, but this problem has not been solved - Jeffreys prior is bad here. $\endgroup$ Oct 10, 2012 at 11:06
  • 3
    $\begingroup$ This theory is incomplete and depends on parameter ordering, choice of compact region, and the likelihood function. So it doesn't obey the likelihood principle for example. Also, it is difficult to apply to non-independent data. Further, Bernardo's theory is only complete for 1-d parameter problems. It is probably the best method available currently though. A good competitor is the transformation group approach of Jaynes. $\endgroup$ Oct 10, 2012 at 22:39
  • 1
    $\begingroup$ @Zen 's comment to Stephane clarifying what invariance means was really helpful when I was trying to understand this concept, so I expanded upon it here: math.stackexchange.com/questions/210607/… $\endgroup$
    – RMurphy
    Aug 3, 2017 at 20:21
52
$\begingroup$

Suppose that you and a friend are analyzing the same set of data using a normal model. You adopt the usual parameterization of the normal model using the mean and the variance as parameters, but your friend prefers to parameterize the normal model with the coefficient of variation and the precision as parameters (which is perfectly "legal"). If both of you use Jeffreys' priors, your posterior distribution will be your friend's posterior distribution properly transformed (don't forget that Jacobian) from his parameterization to yours. It is in this sense that the Jeffreys' prior is "invariant"

(By the way, "invariant" is a horrible word; what we really mean is that it is "covariant" in the same sense of tensor calculus/differential geometry, but, of course, this term already has a well established probabilistic meaning, so we can't use it.)

Why is this consistency property desired? Because, if Jeffreys' prior has any chance of representing ignorance about the value of the parameters in an absolute sense (actually, it doesn't, but for other reasons not related to "invariance"), and not ignorance relatively to a particular parameterization of the model, it must be the case that, no matter which parameterizations we arbitrarily choose to start with, our posteriors should "match" after transformation.

Jeffreys himself violated this "invariance" property routinely when constructing his priors.

$\endgroup$
5
  • 2
    $\begingroup$ +1: Good answer. But, why doesn't the Jeffreys' prior represent ignorance about the value of the parameters? $\endgroup$
    – Neil G
    Oct 9, 2012 at 5:50
  • 4
    $\begingroup$ Because it is not even a distribution. It is paradoxical to claim that a distribution reflects ignorance. A distribution always reflects information. $\endgroup$ Oct 9, 2012 at 6:21
  • 2
    $\begingroup$ Another reference: projecteuclid.org/… $\endgroup$ Oct 9, 2012 at 6:22
  • $\begingroup$ @StéphaneLaurent: One must have some belief even in a state of total ignorance. Whatever your posterior is minus whatever likelihood is induced by your data is the belief that you are assuming in that state of ignorance. The intuitive principle that must be respected when deciding that belief is that it should be invariant under changes of labels (including reparametrization). I'm not sure, but I think that principle alone (in all its possible interpretations — maximum entropy, invariant reparametrization, etc.) always decides the belief. $\endgroup$
    – Neil G
    Jan 30, 2015 at 7:24
  • $\begingroup$ Therefore, when one says "a distribution reflects ignorance", one means that the distribution concords with this principle. $\endgroup$
    – Neil G
    Jan 30, 2015 at 7:25
17
$\begingroup$

To add some quotations to Zen's great answer: According to Jaynes, the Jeffreys prior is an example of the principle of transformation groups, which results from the principle of indifference:

The essence of the principle is just: (1) we recognize that a probability assignment is a means of describing a certain state i knowledge. (2) If the available evidence gives us no reason to consider proposition $A_1$ either more or less likely than $A_2$, then the only honest-way we can describe that state of knowledge is to assign them equal probabilities: $p_1=p_2$. Any other procedure would be inconsistent in the sense that, by a mere interchange of the labels $(1, 2)$ we could then generate a new problem in which our state of knowledge is the same but in which we are assigning different probabilities…

Now, to answer your question: “Why wouldn't you want the prior to change under a change of variables?”

According to Jaynes, the parametrization is another kind of arbitrary label, and one should not be able to “by a mere interchange of the labels generate a new problem in which our state of knowledge is the same but in which we are assigning different probabilities.”

$\endgroup$
8
  • 3
    $\begingroup$ Jaynes seems somewhat mystic to me. $\endgroup$ Oct 9, 2012 at 6:24
  • 1
    $\begingroup$ @StéphaneLaurent: Maybe I was too easily converted then! But, I found this very convincing: E. T. Jaynes, “Where do we stand on Maximum Entropy?,” in The Maximum Entropy Formalism, R. Levine and M. Tribus, Eds. Cambridge, MA, USA: The MIT Press, 1979, pp. 15–118. $\endgroup$
    – Neil G
    Oct 9, 2012 at 6:25
  • 3
    $\begingroup$ Xian received a mail praising Jaynes: ceremade.dauphine.fr/~xian/critic.html It's a pity if you don't read French, this mail is both frightening and funny. The writer seems to have gone crazy by thinking too much about Bayesian statistics ;) $\endgroup$ Oct 9, 2012 at 6:34
  • 2
    $\begingroup$ @StéphaneLaurent: Reading it now. This is absolutely right: "si vous affirmez en page 508 "the nonrepeatability of most experiments" à quoi bon ensuite "looking for optimal fequentist procedures" en page 512? Si la plupart des problèmes ne peuvent donc pas être traités par les procédures fréquentistes, comment le "choix Bayésien", qui se veut être le paradigme pour tout problème inférentiel, n'est-ce pas, peut-il se baser sur une réconciliation avec le fréquentisme (p. 517-518)? Pourquoi ne pas dire une fois pour toute qu'une probabilité n'est jamais une fréquence!" $\endgroup$
    – Neil G
    Jan 30, 2015 at 7:11
  • 2
    $\begingroup$ Also: "Le Principe du Maximum d'Entropie est lui absolument fondamental étant donné qu'il est nécessaire et suffisant pour régler ces cas d'école et que par conséquent il procure dans ces cas la signification véritable des probabilités a priori. Quand on sait qu'il permet ensuite d'unifier Théorie de l'Information, Mécanique Statistique, Thermodynamique…" describes my position as well. However, unlike the writer I have no interest in devoting hours convincing others to accept what I find so natural. $\endgroup$
    – Neil G
    Jan 30, 2015 at 7:18
7
$\begingroup$

While often of interest, if only for setting a reference prior against which to gauge other priors, Jeffreys priors may be completely useless as for instance when they lead to improper posteriors: this is for instance the case with the simple two-component Gaussian mixture $$p\mathcal{N}(\mu_0,\sigma_0^2)+(1-p)\mathcal{N}(\mu_1,\sigma_1^2)$$ with all parameters unknown. In this case, the posterior of the Jeffreys prior does not exist, no matter how many observations are available. (The proof is available in a recent paper I wrote with Clara Grazian.)

$\endgroup$
1
$\begingroup$

If I understood correctly, Jeffrey's prior is motivated in two ways:

  1. The "informativness" of the prior should not differ for different parametrizations of the same prior. E.g. when modelling birth rate of sexes (male vs. female) as binomial, as a "non-informative" prior for the parameter $p$ (probability of child being born as male) one could think of the uniform distribution on $[0, 1]$. But if you would want to consider the odds $\psi=p/(1-p)$, the same prior reparametrized in terms of the odds $\psi$, would not be uniform anymore (thus allocating some odds a higher probability than others. (In more detail: Odds of $3:1$ are now more likely than odds of $5:1$ according to the prior, thus the prior is now (more) "informative" in some way.)

  2. No matter the parametrization of the prior, the resulting posterior (thus the results of an experiment) should only be a monotone transformation of each other. E.g. when researcher $A$ models the prior using parameter $p$ and researcher $B$ models the same prior using parameter $\psi$, they should obtain the same results (up to transformation) when given the same data. Which would be handy if $A$ ran the experiment using $p$ but $B$ is more interested in the odds $\psi$, $B$ could take the results obtained by $A$ and just transform.

$\endgroup$
-4
$\begingroup$

Jeffreys prior is useless. This is because:

  1. It just specifies the form of the distribution; it does not tell you what its parameters should be.
  2. You are never completely ignorant - there is always something about the parameter that you know (e.g. often it cannot be infinity). Use it for your inference by defining a prior distribution. Don't lie to yourself by saying that you don't know anything.
  3. "Invariance under transformation" is not a desirable property. Your likelihood changes under transformation (e.g. by the Jacobian). This does not create "new problems," pace Jaynes. Why shouldn't the prior be treated the same?

Just don't use it.

$\endgroup$
1
  • 4
    $\begingroup$ Eh? Likelihood is not a density and won’t change under reparametrization $\endgroup$
    – innisfree
    Jan 18, 2019 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.