I have two (sparse) large matrices (~ 1million by 1million) and want to compute Mantel statistic to find correlation between them. To counter memory problems, I have computed the mantel statistic between their submatrices by random sampling. I now have a set of Mantel statistic and P-value scores for these pairs. How can I compute the final Mantel statistic between the two original matrices given these bootstrap estimates? Can I compute their mean? This be a trivial question, but I am not a statistician and any help in this regard is much appreciated. Thanks.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
For the Mantel test see here: https://en.wikipedia.org/wiki/Mantel_test
To be able to store two square matrices about the size 1e6 (number of rows and columns) you need storage in the terabyte range, so I doubt you have enough ram. But given that you have enough disk storage, you can compute this via online algorithms, probably indirectly via online algorithms for covariance and variance, see for instance Online algorithm for the mean square error and search this site for tag [online].
That would obliterate the need for putting together results from many random submatrices, although that would be a possibility too. But I leave details of such an approach to others.
answered Aug 9, 2018 at 11:50