Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite

I have two large (sparse) matrices (500000*500000) and would like to perform the mantel test to get a similarity measure.

  • Is there a way to break-up these matrices into smaller ones and perform the test?
  • Or can you suggest a way to do it efficiently without running into memory problems?
share|improve this question
If you just want to estimate a "similarity measure," you don't need the Mantel test: just compute (say) the Pearson correlation between the two matrices. – whuber♦ Oct 17 '11 at 16:18

migrated from stackoverflow.com Oct 7 '11 at 0:37

1 Answer

up vote 1 down vote

You can simply take random samples (i.e. submatrices) and estimate the Mantel test statistic. With enough samples, this should converge on the same conclusion. In this way, you can avoid having to cook up a method for accessing all of the data.

However, as these are sparse matrices, it's better to go after a method for accessing the data via coordinate object lists (COO representation), i.e. (row, column, value), and implement a simple calculation for tabulating the test statistic across the two matrices. You can merge the two matrices' COO versions or perform more complex tabulations using the data.table package.

Ultimately, it depends on how many values exist in your matrices (i.e. how dense they are). Attacking them in sparse matrix formats may be best if they're very sparse. Using a subsampling method may be better if they're not super sparse.

share|improve this answer
Great! I now have the mantel statistic for various pairs of submatrices. To approximate the final correlation, do I compute the mean of these mantel statistics? Also, how do I know the final P-value for this statistic? [please excuse my ignorance] – user975964 Oct 6 '11 at 17:49
Well, this is where the question veers more toward the statistics.SE site... bootstrap estimates of test statistics are a stats topic, rather than a programming topic, so that audience is more apropos. – Iterator Oct 6 '11 at 18:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.