# Significance of parameterisation invariance of Jeffreys prior

I often hear it said that the Jeffreys prior is well-motivated because it is invariant under reparametrization. The proof of this is quite straight-forward (I know the proof on e.g., wiki). I'm a bit confused about what the proof really means, though, because the kind of invariance proven is a bit strange to me. It is indeed proven that if $$p(x) \propto \sqrt{I(x)}$$ then $$p(y) \propto \sqrt{I(y)}$$ where $$I$$ is the Fisher information and $$y$$ was found through a bijective transformation of $$x$$. Note well that $$I(x)$$ is an abuse of notation, as it contains derivatives wrt the variable $$x$$.

I don't see this as particularly compelling, since I make a similar argument that any choice of prior is parametrization invariant. E.g., by writing an arbitrary prior as $$p(\theta) = \frac{dF(\theta)}{d\theta}$$ where $$F$$ is the cumulative distribution function, we then find $$p(\phi) = \frac{dF(\phi)}{d\phi}$$ To put it another way, I can specify a prior by specifying a cdf rather than a pdf, and the cdf transforms trivially under reparameterizations. This kind of invariance is of basically no interest to me.

So, why do people make a fuss about the Jeffreys prior being invariant under reparameterization? I think I would rather say that the kind of invariance that the Jeffreys prior has is necessary for any objective formal rule for selecting a prior, but not in itself a motivation for using a Jeffreys prior. And I think it would be better to say that the Jeffreys rule for making a prior was parameterisation invariant, than say the Jeffreys prior was parameterisation invariant. Is that fair?

• The question is how do you select the distribution $F$? May 6, 2019 at 7:34
• @Xi'an For sure $F$ is totally arbitrary. I'm not suggesting this as a new rule for making priors. That was just an illustration that any prior can be written in this 'invariant' way, such that I don't see it as an advantage of the Jeffreys prior May 6, 2019 at 7:37
• I may have asked the same (or at least a similar) question a few years ago - if so, please check the excellent answers: stats.stackexchange.com/questions/139001/… May 6, 2019 at 19:22
• Thank you, certainly related but I don’t think it’s a duplicate May 6, 2019 at 23:26
• What you are proposing is a tautology. However, Jeffrey's prior is not a tautology. Suppose the data $X \sim f(x; \theta)$. Jeffrey's prior allow you to specify a prior for $f(\theta)$ in terms of $f(x)$, not $F(\theta)$. Mar 24, 2020 at 8:13

I think I see your point. If you define a prior probability density by $$\pi_{\theta}(x) = \frac{d}{d\theta}F(\theta)$$ then under (nice) reparametrization $$\lambda(\theta)$$, you get that that $$\pi_{\lambda}(\lambda) = \frac{d}{d\lambda}F(\theta(\lambda))$$. So the definition is parametrization invariant.

However, this property is nice only if this definition of the prior as $$\frac{d}{d\theta}F(\theta)$$ has a nice meaning, then, this nice meaning is conserved under reparametrization. That's not the case for lots of priors. A uniform distribution is usually meaningfull through its pdf: it's the same everywhere, and this meaning is not stable under reparametrization.

Jeffreys' prior definition is not only parametrization-invariant but also meaningful for a given conditional model. A nice way of seeing this meaning (from Jeffreys (1945)) is to note that if you have conditional densities $$\{f(x|\theta) ; \theta\in\Theta\}$$, then $$KL(f(.|\theta) + f(.|\theta')) \approx I(\theta)(\theta' - \theta)^2$$ if $$\theta$$ and $$\theta'$$ are close (here KL denotes the symmetrized Kullback-Liebler divergence). Then, computing probabilities with Jeffreys' prior corresponding to the conditional model:

$$P([\theta, \theta + d\theta]) = \sqrt{I(\theta)}d\theta = \sqrt{KL[f(.|\theta), f(.|\theta + d\theta})].$$ And thus : $$P([\theta + d\theta]) = P([\theta' + d\theta]) \Leftrightarrow KL\left[f(.|\theta), f(.|\theta + d\theta)\right] = KL\left[f(.|\theta'), f(.|\theta' + d\theta)\right]$$ And this relation holds for any parametrization. So one could say that Jeffreys' prior is uniform in the space of the conditional densities with KL metric (this is informal, KL is not a proper metric). In its original argument, Jeffreys sees this prior as curvilinear coordinate on the map $$\theta \mapsto f(.|\theta)$$ for KL distance, which (I think) is equivalent.

Other more pragmatical arguments for Jeffreys prior could be :

• It often corresponds to already well used objective priors : uniform for a location parameter, inverse for a location parameter, Haar priors.

• It gives bias reduction on the Maximum A Posterior (considered as a frequentist estimator) on exponential family models (see Firth, 1993: Bias Reduction of Maximum Likelihood Estimates).

Hope this brings something..