13
$\begingroup$

I came across a large body of literature which advocates using Fisher's Information metric as a natural local metric in the space of probability distributions and then integrating over it to define distances and volumes.

But are these "integrated" quantities actually useful for anything? I found no theoretical justifications and very few practical applications. One is Guy Lebanon's work where he uses "Fisher's distance" to classify documents and another one is Rodriguez' ABC of Model Selection… where "Fisher's volume" is used for model selection. Apparently, using "information volume" gives "orders of magnitude" improvement over AIC and BIC for model selection, but I haven't seen any follow up on that work.

A theoretical justification might be to have a generalization bound which uses this measure of distance or volume and is better than bounds derived from MDL or asymptotic arguments, or a method relying on one of these quantities that's provably better in some reasonably practical situation, are there any results of this kind?

$\endgroup$
  • $\begingroup$ Fisher's information gives lower bound in parameter estimation. It is a natural metric because it roughly says something like "in this direction the difficulty of my problem cannot decrease more than that". What you call generalization bounds are upper bounds ? do you want to know the performence of the method that use Fisher metric (the large body you mention is a good list)? sorry but I don't really get the question :) can you reformulate that point ? $\endgroup$ – robin girard Aug 13 '10 at 8:45
  • $\begingroup$ Let's say that Fisher's Information Matrix gives our Riemannian metric tensor. It lets us find arclength of any curve by integrating. Then you define the distance between p and q as the smallest arclength over all curves connecting p and q. This is the distance measure I'm asking about. Same with volume. $\endgroup$ – Yaroslav Bulatov Aug 13 '10 at 15:19
  • 1
    $\begingroup$ So, just as an example, Rodriguez gets a significant improvement by using "information volume" as measure of model complexity, but surprisingly I can't see anyone else trying this out $\endgroup$ – Yaroslav Bulatov Aug 16 '10 at 0:54
5
$\begingroup$

There was a read paper last week at the Royal Statistical Society on MCMC techniques over Riemann manifolds, primarily using the Fisher information metric: http://www.rss.org.uk/main.asp?page=1836#Oct_13_2010_Meeting

The results seem promising, though as the authors point out, in many models of interest (such as mixture models) the Fisher information has no analytic form.

$\endgroup$
  • 1
    $\begingroup$ Is that the "Riemann manifold Langevin" paper? Do integrate Fisher information at some point? $\endgroup$ – Yaroslav Bulatov Oct 19 '10 at 21:51
4
$\begingroup$

The most well know argument is that the fisher metric, being invariant to coordinate transforms, can be used to formulate an uninformed prior (Jeffreys prior). Not sure I buy it!

Less well known, is that sometimes these "integrated quantities" turn out to be divergences and such, one may argue that the fisher distances generate a generalised set of divergences (and properties thereof).

But still, I'm yet to find a good intuitive description of the fisher information and the quantities it generates. Please tell me if you find one.

$\endgroup$
  • $\begingroup$ Lot's of things are known about Fisher Information, it's integrals of fisher information that I'm not sure about. I'm not familiar with what you say about Fisher Information turning into some known divergence on integration $\endgroup$ – Yaroslav Bulatov Sep 18 '10 at 14:43
4
$\begingroup$

The reason that there is "no follow up" is that very few people understand the work of Rodriguez on this going back many years. It's important stuff and we will see more of it in the future I am sure.

However, some would argue that the Fisher metric is only a 2nd order approximation to the true metric (e.g. Neumann's paper on establishing entropic priors ) which is actually defined by the Kullback-Liebler distance (or generalisations thereof) and which leads to Zellner's formulation of MDI priors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.