Is there any point to Reverse Engineering the Fisher Information Matrix from an Inverse Covariance Matrix?

Question

Would there be any advantage in deriving a Fisher Information Matrix backwards from an inverse covariance matrix? I've discovered that this is much easier to do on the SQL Server platform I use than calculating Fisher Information in the usual direction, which is not really conducive to SQL solutions; the whole realm of parameter estimation and Maximum Likelihood Estimation (MLE) seems better suited to ordinary computer languages (such as VB.Net or C# on my platform) or packages like R. I would nevertheless like to leverage SQL in my workflow though if there's any practical benefit in doing so, since it is trivial to derive large covariance matrices and invert them in a matter of seconds, on large datasets consisting of tens of millions of rows. Datasets closer to "Big Data" size are the norm, whereas those in ordinary statistical studies (particularly in medicine) seem to run somewhere between a few dozen to a few thousand. I'm basically approaching the question from an unusual vantage point, due to the idiosyncrasies of the platform I use, which allow me to easily fill in the entries of the Fisher Information in reverse with hard numbers, based on the weight of millions or even billions of records.

I've learned from sources like David Wittman's Fisher Information for Beginners that covariance matrices are sometimes constructed for Bayesian priors, then inverted into Fisher Information Matrices and added to other Fisher Information Matrices in order to capture more information in experimental design; the resulting matrix is then inverted in the normal direction to derive the covariance estimates and Cramér-Rao Lower Bound and all that. What I am doing is something different: taking covariance figures calculated from millions of rows and inverting them, which gives me hard numbers to reverse engineer the Fisher Matrix from. By removing some power operations and canceling out some terms, I can basically arrive at the derivatives of both ℒ and the variables. What I need to know is whether the hard numbers this exercise spits out lend themselves to misinterpretation or are even merely useless gibberish. Information measures of all kinds are riddled with all sorts of subtle caveats in interpretation, so I don't want to make any assumptions, especially since I'm still a newbie at Fisher Information.

I know from threads like this one at Quora that the entries of an inverse covariance matrix have value in and of themselves, in the form of partial correlations. The CrossValidated thread What Does the Inverse of Covariance Matrix Say About Data? (Intuitively) also discusses many other useful properties and mentions in passing that it's equivalent to the Fisher Matrix. How to interpret an inverse covariance or precision matrix? is another useful one that gets into partial correlations.

My question(s) would be:

1. Is there any point to doing this? My main concern would be that if we can derive exact covariances already, there might not be any sense in working backwards to finding something equivalent to likelihood estimates. Why do estimation if we already have certainties?

2. Would the numbers returned have any validity (as opposed to usefulness, which is a different matter)?

3. What meaning and mathematical properties would the entries of a Fisher Information Matrix reverse-engineered in this way have? Would there be any difference from ordinary Fisher Information? This would be a crucial consideration.

4. A tangential question of less relevance: Do any of the considerations above change if the millions or billions of rows I'm plugging in are merely a sample of some unknown full population of billions or trillions? What if the proportional difference between the two is quite narrow - say, for example, the full population is only two or three times the size of our "Big Data"-sized sample? In the database server world, it seems to be more common to work with datasets that are proportionally closer or even identical to the full population.

If these are dumb questions, I'll accept an answer that explains why. I've never seen these considerations addressed in the literature I've skimmed over the past few years, so either my approach is so ludicrous that it goes without saying that it wouldn't work, or there's some merit to it, but only in the kind of unusual use cases I run into all the time.

Answer 1

Generally, the answer is no, reverse engineering the Fisher Information Matrix from a covariance matrix would not be useful even in many extraordinary cases. My reasoning when posting this question was fallacious in several ways, which only became obvious when I actually walked through the process while attempting to write code for this informal blog post.

• The main issue was what I call “definition drift”: gradually and unconsciously altering the interpretation of a stat or model over the course of many operations or logical steps. The covariance matrices that I referred to as being much easier to calculate on relational databases are on the actual data points, so each diagonal would represent the variance of a particular variable (or database column). If we’re inverting Fisher Information Matrices for Maximum Likelihood Estimates (MLEs), then we’re instead calculating the variances of statistics about those variables (or aggregates on columns in database parlance). This is an apples vs. oranges comparison. The same would be true if we tried to reverse engineer Fisher Information from another candidate, like the Kullback-Leibler Divergence or Shannon-Jensen Divergence.

• The second and third questions in the list became self-evident when I attempted to walk through the reverse engineering process step-by-step. The only way to validate the results of the covariance matrix-first approach would have been to make sure they matched up with the entries of a Fisher Information Matrix derived in the ordinary manner. If the numbers and the properties of the resulting matrix weren’t the same then the exercise would have been invalid. I came up with this test but never executed it because of the first problem in this list, but it really is a matter of common sense. Deriving a covariance matrix from the data points instead of aggregates upon would definitely have produced different results.

• Even if all of the above weren’t true, ordinary confidence interval testing and methods of determining the bias of estimators in advance would have been able to determine whether or not “Big Data” rowcounts obviate the need for more records/trials. One of the major uses of MLE is in optimizing experiment design, precisely to determine just how much data is needed to analyze a particular model, via asymptotic estimators and reliance on the Cramer-Rao Bound. That still leaves an interesting side question unanswered: has the era of Big Data close the gap somewhat between sample sizes and the overall populations we’re comparing them to, in general? If a table has one entry apiece for a billion living individuals, we might be able to make solid inferences about the whole human race, since there are only 7 billion+ of us; if we have a billion rows of cell data, that’s a drop in the bucket compared to the estimated 28.5 septillion human cells on the planet. To my knowledge, the effect of Big Data on such gaps has not been formally studied. If the gap has closed, then we can put more reliance than in the past on simple aggregates taken over Big Data database tables, which are trivial to calculate even on a billion rows, obviating the need for MLE/Fisher techniques. I would assume the gap has closed somewhat, given that most tutorials I’ve seen on Fisher Information mention plugging in just a few hundred records at most.

There may be exceptions to the rules above. For example, Fisher Information can be used to calculate the Jeffreys Prior or derive a complicated but interesting Minimum Description Length criterion for judging mining model quality, one taking into account goodness-of-fit, dimensionality and geometric complexity simultaneously.1 There are probably many others. It might conceivably be possible to plug the Fisher Information calculated on variables into these formulas in place of the Fisher Information of aggregates on those variables; I have yet to attempt this directly. The kicker is that even in these uncommon use cases, it would be much simpler to just take the reciprocal of the variance of each variable, which is trivial to calculate; resorting to covariance matrix inversion would be an unnecessary and expensive step. Perhaps there is a use out there for reverse engineering Fisher Information from covariance matrices, but it seems to be inapplicable in even the most exotic use cases known today (Wittman’s method excepted, perhaps).

1 p. 29, Ly, Alexander; Verhagen, Josine; Grasman, Raoul and Wagenmakers, Eric-Jan, 2014, “A Tutorial on Fisher Information,” published on the personal webpage of Eric-Jan Wagenmakers at http://www.ejwagenmakers.com/submitted/LyEtAlTutorial.pdf I have not had a chance to read the latest version published in 2017 at arXiv.org.