I am trying to test whether the covariance matrix for the maximum likelihood estimates for a gaussian general linear model approaches the inverse Fisher information matrix (times 1/n , n being the sample size).

However I am not sure how to compare two matrices. One approach would be to compare the eigenvalues. However I am finding with my runs of code, that the eigenvalues of my covariance matrix tend to vary quite wildly. I have tried using the spectral norm (i.e largest eigenvalue, but it is often the case that I have a single very large eigenvalue (say, order unity), and the others are much smaller (of order 10e-15). The Fisher information matrix eigenvalues tend to be quite consistent (about 10e-6, following the example with the same number of samples, observations and covariates for comparison).

So how would one compare two matrices, and relatively easily? Is there a statistical test for this?

My thoughts are that:

-IF we work in a given basis, then the features defining a matrix are:

  1. It's rank

  2. The eigenvectors (and thus the subspace spanned by them)

  3. The eigenvalues

The simplest case for comparison would be if the rank and eigenvectors were the same, so only the eigenvalues had to be compared. Otherwise, depending on what one means be 'how different the matrices are', one would have to have a metric that accounts for the difference in the eigenvalues (perhaps a measure of the overlap of the subspace spanned by them? But this doesn't account for all of the information!), and the eigenvalues. Perhaps also whether the eigenvalues correspond to the same vectors.

So I see that this is not straightforward and probably depends on what you want to do with the information. But in that case, how to meaningfully compare matrices? I suppose in this case of MLE, we really mean element-wise comparison.

  • $\begingroup$ Have you thought about measuring the dissimilarity between the two matrices using some matrix norm of their difference ($\|A-B\|$)? More details about your problem would be needed to think about a statistical test. For example, how are you obtaining these matrices? Where does any kind of randomness/variability come into play that would let us think about how to define a null distribution? $\endgroup$ – user20160 Jul 18 '19 at 16:57
  • 1
    $\begingroup$ See stats.stackexchange.com/a/469966/919 for one recently proposed approach. $\endgroup$ – whuber Dec 30 '20 at 14:20

The answer depends on where these matrices come from. In latent variable modeling techniques this is commonly done. For example, in Structural Equation Models the loss function for maximum likelihood estimated models is:

$$\text{Fml} = \text{log}|\Sigma(0)| + \text{Trace}[\Sigma(0)^{-1}S] - \text{log}|S| - p$$

Maximizing the Fml minimizes the discrepancy between the estimated population covariance matrix $\Sigma$, and the model implied covariance matrix $\Sigma(0)$. The population covariance is estimated as a function of the observed sample covariance matrix $S$. Model fit statistics provide summaries of the discrepancies between these two matrices. David Kenny provides brief summaries of these statistics but cites the work necessary to break them down further. Many model fit statistics exist which compare these matrices in different ways, you may be able to conform your situation to these.

To answer your question completely, more information would be needed regarding where they come from. However, this literature may give you a good starting point on exploring what options exist.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.