In our project, we would like to do some optimization on sparse matrices. The idea is to scrape massive amounts of data, form a covariance/correlation matrix, and form a sparsity pattern basically by thresholding the values and then correcting them to be positive semidefinite / with unit diagonal. We are fairly certain this is a useful thing to do, but I am personally very vague as to the specifics.
So, here are my questions.
1) What is the difference in usage between a covariance and correlation matrix? I get what the definitions are, but in what scenario would a sparsity pattern generated by thresholding covariance over correlation make sense, and vice versa? (It seems like if the data may have massively different magnitudes, thresholding correlation makes more sense, but most papers seem to focus on covariance.) What does it mean to have a strong covariance but weak correlation, or vice versa?
2) What is an appropriate threshold on a sample covariance/correlation matrix? Specifically, under what scenarios would it be appropriate to threshold say, all correlation values with absolute value less than 0.3 to 0? I'm looking at some stock price time series, and two seemingly unrelated stocks can very easily have a correlation of 0.3, so I feel like that should be appropriate, but I can't say why. I saw some sources on what is a threshold for significant data above a certain tail measure, but these values are all very small. (For x samples, there is a y probability that I will get to z tail value, and always z is very small.) I tried doing a permutation test, but I am not sure exactly how to interpret the results. Say that, for the randomly permuted correlation, I get a value of 0.8 in the 90th percentile. Does this mean it's appropriate to threshold the correlation matrix at abs = 0.8? That seems high. Also, saying that the data for two variables don't conclusively suggest correlation does not mean the true correlation value is 0. So overall this concept confuses me. I know some people use L1 regularization, but even then you have to pick a weight on the regularization term; what's the metric you use to decide what sparsity is appropriate?
Edit: I'm looking at Bickel/Levina 2008 (section 3.1) which proposes a method to pick a threshold (for covariance matrices only), but it is a very convoluted method, and I keep getting a recommended threshold of 0 (aka no thresholding), even when the number of samples is very low. This seems weird. Is this really what is done in practice?
3) What do people do with these sparse thresholded matrices? Why are they preferred over simply C = XX.T (where X is a demeaned sample data matrix)? Is it simply because sparse matrix is a better estimation or simply because of storage reasons, or are there specific applications (I guess related to graphical representations?) where a sparsity pattern is important?
I'm also not a statistics person so I'd definitely appreciate referrals to useful references. Thanks for any discussion!