In our project, we would like to do some optimization on sparse matrices. The idea is to scrape massive amounts of data, form a covariance/correlation matrix, and form a sparsity pattern basically by thresholding the values and then correcting them to be positive semidefinite / with unit diagonal. We are fairly certain this is a useful thing to do, but I am personally very vague as to the specifics.

So, here are my questions.

1) What is the difference in usage between a covariance and correlation matrix? I get what the definitions are, but in what scenario would a sparsity pattern generated by thresholding covariance over correlation make sense, and vice versa? (It seems like if the data may have massively different magnitudes, thresholding correlation makes more sense, but most papers seem to focus on covariance.) What does it mean to have a strong covariance but weak correlation, or vice versa?

2) What is an appropriate threshold on a sample covariance/correlation matrix? Specifically, under what scenarios would it be appropriate to threshold say, all correlation values with absolute value less than 0.3 to 0? I'm looking at some stock price time series, and two seemingly unrelated stocks can very easily have a correlation of 0.3, so I feel like that should be appropriate, but I can't say why. I saw some sources on what is a threshold for significant data above a certain tail measure, but these values are all very small. (For x samples, there is a y probability that I will get to z tail value, and always z is very small.) I tried doing a permutation test, but I am not sure exactly how to interpret the results. Say that, for the randomly permuted correlation, I get a value of 0.8 in the 90th percentile. Does this mean it's appropriate to threshold the correlation matrix at abs = 0.8? That seems high. Also, saying that the data for two variables don't conclusively suggest correlation does not mean the true correlation value is 0. So overall this concept confuses me. I know some people use L1 regularization, but even then you have to pick a weight on the regularization term; what's the metric you use to decide what sparsity is appropriate?

Edit: I'm looking at Bickel/Levina 2008 (section 3.1) which proposes a method to pick a threshold (for covariance matrices only), but it is a very convoluted method, and I keep getting a recommended threshold of 0 (aka no thresholding), even when the number of samples is very low. This seems weird. Is this really what is done in practice?

3) What do people do with these sparse thresholded matrices? Why are they preferred over simply C = XX.T (where X is a demeaned sample data matrix)? Is it simply because sparse matrix is a better estimation or simply because of storage reasons, or are there specific applications (I guess related to graphical representations?) where a sparsity pattern is important?

I'm also not a statistics person so I'd definitely appreciate referrals to useful references. Thanks for any discussion!

  • 1
    $\begingroup$ Try some more principled method, like lasso? See faculty.bscb.cornell.edu/~bien/papers/biometrika2011spcov.pdf there are a lot of papers! $\endgroup$ – kjetil b halvorsen Sep 10 '17 at 11:51
  • 2
    $\begingroup$ @kjetilbhalvorsen that's an interesting paper! The first author wrote another paper in this area last year (see arxiv.org/pdf/1606.00451.pdf) There's an interesting framework provided in the first few pages to help understand the previous work in the area. The paper also proposes a very interesting generalization of much existing work. $\endgroup$ – user795305 Sep 11 '17 at 15:19
  • $\begingroup$ Thanks for the links! Our method is actually a different thing we hope to compare against graphical LASSO, under different assumptions. (I guess this is a common trend these days.) It will be useful to use their evaluation methods to see how well each method does, thanks! $\endgroup$ – Y. S. Sep 11 '17 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.