I am reposting an "answer" to a question that I had given some two weeks ago here: Why is the Jeffreys prior useful? It really was a question (and I did not have the right to post comments at the time, either), though, so I hope it is OK to do this:

In the link above it is discussed that the interesting feature of Jeffreys prior is that, when reparameterizing the model, the resulting posterior distribution gives posterior probabilities that obey the restrictions imposed by the transformation. Say, as discussed there, when moving from the success probability $\theta$ in the Beta-Bernoulli example to odds $\psi=\theta/(1-\theta)$, it should be the case that the a posterior satisfies $P(1/3\leq\theta\leq 2/3\mid X=x)=P(1/2\leq\psi\leq 2\mid X=x)$.

I wanted to create a numerical example of invariance of Jeffreys prior for transforming $\theta$ to odds $\psi$, and, more interestingly, lack thereof of other priors (say, Haldane, uniform, or arbitrary ones).

Now, if the posterior for the success probability is Beta (for any Beta prior, not only Jeffreys), the posterior of the odds follows a Beta distribution of the second kind (see Wikipedia) with the same parameters. Then, as highlighted in the numerical example below, it is not too surprising (to me, at least) that there is invariance for any choice of Beta prior (play around with alpha0_U and beta0_U), not only Jeffreys, cf. the output of the program.

library(GB2) 
# has the Beta density of the 2nd kind, the distribution of theta/(1-theta) if theta~Beta(alpha,beta)

theta_1 = 2/3 # a numerical example as in the above post
theta_2 = 1/3

odds_1 = theta_1/(1-theta_1) # the corresponding odds
odds_2 = theta_2/(1-theta_2)

n = 10 # some data
k = 4

alpha0_J = 1/2 # Jeffreys prior for the Beta-Bernoulli case
beta0_J = 1/2
alpha1_J = alpha0_J + k # the corresponding parameters of the posterior
beta1_J = beta0_J + n - k

alpha0_U = 0 # some other prior
beta0_U = 0
alpha1_U = alpha0_U + k # resulting posterior parameters for the other prior
beta1_U = beta0_U + n - k

# posterior probability that theta is between theta_1 and theta_2:
pbeta(theta_1,alpha1_J,beta1_J) - pbeta(theta_2,alpha1_J,beta1_J) 
# the same for the corresponding odds, based on the beta distribution of the second kind
pgb2(odds_1, 1, 1,alpha1_J,beta1_J) - pgb2(odds_2, 1, 1,alpha1_J,beta1_J) 

# same for the other prior and resulting posterior
pbeta(theta_1,alpha1_U,beta1_U) - pbeta(theta_2,alpha1_U,beta1_U)
pgb2(odds_1, 1, 1,alpha1_U,beta1_U) - pgb2(odds_2, 1, 1,alpha1_U,beta1_U)

This brings me to the following questions:

  1. Do I make a mistake?
  2. If no, is there a result like there being no lack of invariance in conjugate families, or something like that? (Quick inspection leads me to suspect that I could for instance also not produce lack of invariance in the normal-normal case.)
  3. Do you know a (preferably simple) example in which we do get lack of invariance?
  • 1
    You do not need the R code (which I cannot run with R version 3.0.2) to verify the invariance since it is a property of the likelihood. What is meant by prior invariance is the construction of a rule for selecting prior that does not depend on the choice of the parametrisation of the sampling model. – Xi'an Feb 24 '15 at 10:22
  • 1
    I am sorry for the inconvenience. It runs with R 3.1.2 on my computer. If I may follow up, does your comment imply that I misunderstood Zen's comment on the accepted answer, item 1., of Stephane Laurent on Why is the Jeffreys prior useful?? – Christoph Hanck Feb 24 '15 at 10:29
up vote 16 down vote accepted
+100

Your computation seems to be verifying that, when we have a particular prior distribution $p(\theta)$ the following two procedures

  1. Compute the posterior $p_{\theta \mid D}(\theta \mid D)$
  2. Transform the aforementioned posterior into the other parametrization to obtain $p_{\psi \mid D}(\psi \mid D)$

and

  1. Transform the prior $p_\theta(\theta)$ into the other parametrization to obtain $p_\psi(\psi)$
  2. Using the prior $p_\psi(\psi)$, compute the posterior $p_{\psi \mid D}(\psi \mid D)$

lead to the same posterior for $\psi$. This will indeed always occur (caveat; as long as the transformation is such that a distribution over $\psi$ is determined by a distribution over $\theta$).

However, this is not the point of the invariance in question. Instead, the question is whether, when we have a particular Method For Deciding The Prior, the following two procedures:

  1. Use the Method For Deciding The Prior to decide $p_\theta(\theta)$
  2. Convert that distribution into $p_\psi(\psi)$

and

  1. Use the Method For Deciding The Prior to decide $p_\psi(\psi)$

result in the same prior distribution for $\psi$. If they result in the same prior, they will indeed result in the same posterior, too (as you have verified for a couple of cases).

As mentioned in @NeilG's answer, if your Method For Deciding The Prior is 'set uniform prior for the parameter', you will not get the same prior in the probability/odds case, as the uniform prior for $\theta$ over $[0,1]$ is not uniform for $\psi$ over $[0,\infty)$.

Instead, if your Method For Deciding The Prior is 'use Jeffrey's prior for the parameter', it will not matter whether you use it for $\theta$ and convert into the $\psi$-parametrization, or use it for $\psi$ directly. This is the claimed invariance.

It looks like you're verifying the likelihoods induced by the data are unaffected by parametrization, which has nothing to do with the prior.

If your way of choosing priors is to, e.g., "choose the uniform prior", then what is uniform under one parametrization (say Beta, i.e. Beta(1,1)) is not uniform under another, say, BetaPrime(1,1) (which is skewed) — it's BetaPrime(1,-1) is uniform if such a thing exists.

The Jeffreys prior is the only "way to choose priors" that is invariant under reparametrization. So it is less assumptive than any other way of choosing priors.

  • I do not think the Jeffreys prior is the only invariant prior. When they differ, left and right Haar measures are both invariant. – Xi'an Feb 24 '15 at 10:10
  • @Neil G, I am not sure I can follow your reasoning that I only look at the likelihood. When plugging (e.g.) alpha1_J into pbeta and pgb2 this parameter is determined by both a prior parameter (alpha1_J) and the data (k), likewise for all the other parameters. – Christoph Hanck Feb 24 '15 at 10:22
  • 1
    (+1) You'd hope elicitation of subjective priors would be parametrization-invariant too. – Scortchi Feb 24 '15 at 14:58
  • 1
    @Zen: yes indeed,I was too hasty: Haar measures are a incorrect example. Still, I wonder why Jeffreys' is the only invariant prior... – Xi'an Feb 25 '15 at 8:51
  • 2
    @Xi'an: if my memory doesn't fail me, there is a Theorem in Cencov book (amazon.com/…) which, in some sense (?), proves that Jeffreys prior is the only guy in the town with the necessary invariance. His proof is inaccessible to me. It uses the language of Category Theory, functors, morphisms and all that. en.wikipedia.org/wiki/Category_theory – Zen Feb 25 '15 at 15:46

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.