Suppose I have a large number of features, and I want to figure out which of them (if any) are "significantly" correlated. In short, how I can estimate significance in a way that appropriately accounts for the number of tests I'm doing?

According to Wikipedia, if $r$ is the Pearson correlation over $N$ samples, then the test statistic $t = r\sqrt{\frac{N-2}{1-r ^2}}$ is distributed as a $t_{N-2}$ distribution under the null that true correlation is zero. If I have $D$ features, then I'm looking at $D(D-1)/2$ correlations, so I ought to do some sort of multiple testing correction. But intuitively, these aren't all independent tests ($A \propto B, B\propto C \Rightarrow A \propto C$), so something like a Bonferroni correction would be overly aggressive. Is there a better correction to use?

I've also seen recommendations to use a permutation test, but it's not entirely clear to me how one would do that for a matrix of correlations. But that might be a separate question.

NB: this is related to, but different from, Appropriate Multiple Testing Correction for Correlation Matrix, which asks about testing whether any of the variables are significantly correlated.

  • $\begingroup$ How does your test differ from the question you reference? What is your null hypothesis? What is the alternative hypothesis? $\endgroup$ – whuber Feb 10 '19 at 22:35
  • $\begingroup$ I'm running D*(D-1)/2 tests: For each pair of features, I'm testing the null hypothesis that the correlation is zero vs the alternative that the correlation is nonzero. Whereas the referenced question is testing a single null hypothesis that none of the features are correlated vs a single alternative that any of the features are correlated. $\endgroup$ – Empiromancer Feb 11 '19 at 1:37
  • $\begingroup$ How does that differ from the question you reference, which seems to address exactly the same thing? $\endgroup$ – whuber Feb 11 '19 at 13:05
  • $\begingroup$ @whuber Perhaps I'm not understanding your question. The referenced question asks about a single compound hypothesis for a set of correlations; my question here asks about many individual hypotheses. Sure, they're related (the other question is about the OR of all of the hypotheses here) but I don't see how answering whether any pair of variables in a set are correlated would tell me which ones are (significantly) correlated. Surely that makes it a different question? $\endgroup$ – Empiromancer Feb 11 '19 at 18:55
  • $\begingroup$ I agree it's a different question: thank you for clarifying the distinction. It appears to be exactly the same issue that pertains to ANOVA, where it is solved by first performing an "omnibus" test (usually an F test) of "overall significance," and then, conditional on deciding there exists some difference, following it by individual tests of the coefficients. This suggests the two versions of your question are intimately connected and perhaps can be addressed with the same (or similar) procedures. $\endgroup$ – whuber Feb 11 '19 at 20:07

You can find the algorithm for multiple testing on correlation matrices that accounts for heavy dependence in Cai, T. T., & Liu, W. (2016). Large-Scale Multiple Testing of Correlations. Journal of the American Statistical Association, 111(513), 229–240. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894362/ I haven't seen any implementations of it unfortunately.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.