2
$\begingroup$

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured sparsity (such as banding or hard thresholding) and adjusting it back to a positive semidefinite matrix.

The weird thing is that I am having the hardest time finding real data that actually has sparse covariance! I think the place to look is something with a band pattern, such as with shifted time series, but I can't think of the right application for this. Does anyone have any suggestions?

Either abstract concepts or links to data are appreciated; I'm getting good at finding data once I know what application I'm targeting but I'm blanking out now on what application to use!

Thanks!

$\endgroup$
  • 2
    $\begingroup$ Perhaps you mean sparse inverse covariance matrices? I think in general it will be hard to find sparse covariance matrices unless the noise/error is really tiny. $\endgroup$ – purple51 Mar 26 '15 at 0:57
  • $\begingroup$ That is useful information for me! I was hoping to find maybe something that propagates through time for a banded pattern, like let's say traffic data, but so far no luck; this could be why. Ok let's say sparse inverse covariance matrix. What might work well for that? Thanks for your input! $\endgroup$ – Y. S. Mar 26 '15 at 2:11
  • $\begingroup$ Also this may seem like a stupid question but is a sample inverse covariance matrix calculated by inverting a sample covariance matrix? Or is there a more preferred way? (I'm guessing that would be terribly numerically unstable...) $\endgroup$ – Y. S. Mar 26 '15 at 2:14
  • 1
    $\begingroup$ Just to clarify, not saying that sparse covariance is impossible, just that for all the data I see (biological data), the high degree of correlation means even if the "true" signal is sparse you get non-sparse covariance. To infer sparse inverse covariance from non-sparse covariance is a statistical problem in itself. Like you say, simple inversion can be problematic depending on the condition number, L2-penalized inversion is better but not sparse, but L1-penalized inversion (graphical lasso) can give you sparse inverse covariance. $\endgroup$ – purple51 Mar 26 '15 at 2:20
  • 1
    $\begingroup$ Yes time series or spatial data that has already been detrended are good bets, as are spatially clustered data (e.g. students within the same school may be correlated, but should be uncorrelated between schools). $\endgroup$ – Andy W Mar 26 '15 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.