Significance of parameterisation invariance of Jeffreys prior

Question

I often hear it said that the Jeffreys prior is well-motivated because it is invariant under reparametrization. The proof of this is quite straight-forward (I know the proof on e.g., wiki). I'm a bit confused about what the proof really means, though, because the kind of invariance proven is a bit strange to me. It is indeed proven that if $$ p(x) \propto \sqrt{I(x)} $$ then $$ p(y) \propto \sqrt{I(y)} $$ where $I$ is the Fisher information and $y$ was found through a bijective transformation of $x$. Note well that $I(x)$ is an abuse of notation, as it contains derivatives wrt the variable $x$.

I don't see this as particularly compelling, since I make a similar argument that any choice of prior is parametrization invariant. E.g., by writing an arbitrary prior as $$ p(\theta) = \frac{dF(\theta)}{d\theta} $$ where $F$ is the cumulative distribution function, we then find $$ p(\phi) = \frac{dF(\phi)}{d\phi} $$ To put it another way, I can specify a prior by specifying a cdf rather than a pdf, and the cdf transforms trivially under reparameterizations. This kind of invariance is of basically no interest to me.

So, why do people make a fuss about the Jeffreys prior being invariant under reparameterization? I think I would rather say that the kind of invariance that the Jeffreys prior has is necessary for any objective formal rule for selecting a prior, but not in itself a motivation for using a Jeffreys prior. And I think it would be better to say that the Jeffreys rule for making a prior was parameterisation invariant, than say the Jeffreys prior was parameterisation invariant. Is that fair?

Answer 1

I think I see your point. If you define a prior probability density by $\pi_{\theta}(x) = \frac{d}{d\theta}F(\theta)$ then under (nice) reparametrization $\lambda(\theta)$, you get that that $\pi_{\lambda}(\lambda) = \frac{d}{d\lambda}F(\theta(\lambda))$. So the definition is parametrization invariant.

However, this property is nice only if this definition of the prior as $\frac{d}{d\theta}F(\theta)$ has a nice meaning, then, this nice meaning is conserved under reparametrization. That's not the case for lots of priors. A uniform distribution is usually meaningfull through its pdf: it's the same everywhere, and this meaning is not stable under reparametrization.

Jeffreys' prior definition is not only parametrization-invariant but also meaningful for a given conditional model. A nice way of seeing this meaning (from Jeffreys (1945)) is to note that if you have conditional densities $\{f(x|\theta) ; \theta\in\Theta\}$, then $KL(f(.|\theta) + f(.|\theta')) \approx I(\theta)(\theta' - \theta)^2$ if $\theta$ and $\theta'$ are close (here KL denotes the symmetrized Kullback-Liebler divergence). Then, computing probabilities with Jeffreys' prior corresponding to the conditional model:

$$P([\theta, \theta + d\theta]) = \sqrt{I(\theta)}d\theta = \sqrt{KL[f(.|\theta), f(.|\theta + d\theta})].$$ And thus : $$P([\theta + d\theta]) = P([\theta' + d\theta]) \Leftrightarrow KL\left[f(.|\theta), f(.|\theta + d\theta)\right] = KL\left[f(.|\theta'), f(.|\theta' + d\theta)\right]$$ And this relation holds for any parametrization. So one could say that Jeffreys' prior is uniform in the space of the conditional densities with KL metric (this is informal, KL is not a proper metric). In its original argument, Jeffreys sees this prior as curvilinear coordinate on the map $ \theta \mapsto f(.|\theta)$ for KL distance, which (I think) is equivalent.

Other more pragmatical arguments for Jeffreys prior could be :

• It often corresponds to already well used objective priors : uniform for a location parameter, inverse for a location parameter, Haar priors.

• It gives bias reduction on the Maximum A Posterior (considered as a frequentist estimator) on exponential family models (see Firth, 1993: Bias Reduction of Maximum Likelihood Estimates).

Hope this brings something..