I am exploring ways to reduce the noise of a covariance matrix estimator when the number of variables is greater than the number of observations, i.e. $n > t$.

First, I tried using a low rank estimator of the matrix, constructed from principal components (PC) that account for 90% of the sample variance. I tested this approach by choosing a certain covariance matrix $\Sigma$, which had a highly correlated block and some less correlated variables. I then took samples from a normal distribution with the structure of this covariance matrix $$ X = \Sigma{}^{1/2}N, $$ where $N \sim \mathbb{N}(0,1)$. For every draw, I computed a sample covariance matrix and the low rank PC estimator. Then, I computed the norms of differences between the test matrix and the estimators $||\Sigma-\hat{\Sigma}||$. After collecting a large number or test results, I see that on average the PCA estimator performs noticeably worse that the sample covariance matrix.

This method is often encountered in literature, so I am questioning my approach here... Is there a flaw in the design of my test?

Edit: Through experimentation guided by trial and error I found that if I do PCA on subsets of variables first, build projection matrices from vectors corresponding to largest eigenvalues, then act with those on generated samples of variables in respective subsets and finally compute the covariance matrix, the resulting estimator performs better than the sample covariance matrix. Choice of subsets is naturally dictated by the test data in my case because some variables are highly correlated.

It is interesting that the rank of an estimator constructed in this way is almost the same as of the sample covariance matrix, but statistics of errors are noticeably improved... Has this approach been studied and justified before? If so, does it have a name so I can look into this?

  • 1
    $\begingroup$ What norm did you use? $\endgroup$ – Mark L. Stone Apr 15 '18 at 18:23
  • $\begingroup$ I tried both Frobenius and $L_1$ norms. The results seem to be insensitive to this. $\endgroup$ – J.K. Apr 15 '18 at 20:31
  • $\begingroup$ did you try shrinkage methods already? e.g. Ledoit's method $\endgroup$ – Aksakal Apr 17 '18 at 15:58
  • $\begingroup$ any more info on this issue? interesting problem. i'm seeing similar results (worse performance of low rank covariance matrix than sample covariance) $\endgroup$ – Michael Aug 6 '18 at 18:34
  • $\begingroup$ I see better results using shrinkage methods. I'd like to ideally combine shrinkage methods + low rank covariance because in my mind they have slightly different purposes $\endgroup$ – Michael Aug 6 '18 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.