Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a several hundreds of measurements. Now I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are hundreds of correlations. Among these there should (statistically) be a high correlation, even if the data is completely random (each measure has only about 100 datapoints).

When I find a correlation, how do I include the information about how hard I looked for a correlation into it?

I am not at a high level in statistics, so please bear with me.

share|improve this question
This is a great example of why one needs multiple hypothesis testing. –  mbq Dec 26 '10 at 13:44
Presumably one can use the permutation procedure to generate a null distribution for significance thresholds for the largest correlation, a different threshold for the second largest correlation, and so on. Hopefully this would only take a few hours in Python or R. (Ha! Famous last words.) But surely someone must already have done this and saved the code somewhere? –  user16977 Nov 19 '12 at 0:40
@tmo R on this machine takes 18 seconds to obtain 1000 realizations of the null permutation distribution of the max correlation coefficient for a 300 by 100 matrix x: correl <- function(x, k=1) { n <- dim(x)[2] * (dim(x)[2]-1) / 2; v <- cor(x); sort(v[lower.tri(v)])[(n-k+1):n] }; sim <- replicate(1000, correl(apply(x,2,sample))) –  whuber Nov 19 '12 at 17:45

4 Answers 4

up vote 47 down vote accepted

This is an excellent question, worthy of someone who is a clear statistical thinker, because it recognizes a subtle but important aspect of multiple testing.

There are standard methods to adjust the p-values of multiple correlation coefficients (or, equivalently, to broaden their confidence intervals), such as the Bonferroni and Sidak methods (q.v.). However, these are far too conservative with large correlation matrices due to the inherent mathematical relationships that must hold among correlation coefficients in general. (For some examples of such relationships see the recent question and the ensuing thread.) One of the best approaches for dealing with this situation is to conduct a permutation (or resampling) test. It's easy to do this with correlations: in each iteration of the test, just randomly scramble the order of values of each of the fields (thereby destroying any inherent correlation) and recompute the full correlation matrix. Do this for several thousand iterations (or more), then summarize the distributions of the entries of the correlation matrix by, for instance, giving their 97.5 and 2.5 percentiles: these would serve as mutual symmetric two-sided 95% confidence intervals under the null hypothesis of no correlation. (The first time you do this with a large number of variables you will be astonished at how high some of the correlation coefficients can be even when there is no inherent correlation.)

When reporting the results, no matter what computations you do, you should include the following:

  • The size of the correlation matrix (i.e., how many variables you have looked at).

  • How you determined the p-values or "significance" of any of the correlation coefficients (e.g., left them as-is, applied a Bonferroni correction, did a permutation test, or whatever).

  • Whether you looked at alternative measures of correlation, such as Spearman rank correlation. If you did, also indicate why you chose the method you are actually reporting on and using.

share|improve this answer
Thanks for the excellent answer! –  David Dec 26 '10 at 0:18
(+1) Nice answer! –  chl Dec 26 '10 at 14:08
This is a pretty thorough descriptiion of p-value adjustment methods but what is left unsaid is the criteria for adjustment. Traditionally it has been familywise error rate. But that is a strict criterion and is not useful when you are looking at thousands of comparisons. In that case the false discovery rate first suggested by Benjamini is now commonly used. –  Michael Chernick May 5 '12 at 3:01
What if we just want to look at correlations of very well defined pairs of variables (e.g. $corr(x_1,y_1)$,...,$corr(x_n,y_n)$, where each $x_i$ and $y_i$ are variables) but don't care about all the other possible combinations (i.e., don't care about $corr(x_i,y_j)$ $\forall i \not= j$)? Do we still need a correction? –  Jase Dec 16 '12 at 16:19
@Jase Yes, you do. The amount of correction depends on the interrelationships among the variables. Simulation-based methods are about the only practicable way to determine these corrections. –  whuber Dec 16 '12 at 16:31

From your follow up response to Peter Flom's question, it sounds like you might be better served by techniques that look at higher level structure in your correlation matrix.

Techniques like factor analysis, PCA, multidimensional scaling, and cluster analysis of variables can be used to group your variables into sets of relatively more related variables.

Also, you may want to think theoretically about what kind of structure should be present. When your number of variables is large and the number of observations is small, you are often better relying more on prior expectations.

share|improve this answer
(+1) Good advice in light of the OP's comment to @Peter Flom's response. –  whuber Dec 31 '10 at 5:16

This is an example of multiple comparisons. There's a large literature on this.

If you have, say, 100 variables, then you will have 100*99/2 =4950 correlations.

If the data are just noise, then you would expect 1 in 20 of these to be significant at p = .05. That's 247.5

Before going farther, though, it would be good if you could say WHY you are doing this. What are these variables, why are you correlating them, what is your substantive idea?

Or, are you just fishing for high correlations?

share|improve this answer
The reason why I wanted to do it like this was to have an open mind towards understanding my data, so maybe in a way I am fishing for correlations, which I did not think of before, for the purpose of getting enlightened. I am certainly not doing this to satisfy my boss or something abitrary. I would rather not get into the specifics of the data, as I want a general answer to this question, so I can use it in all situations in the future. –  David Dec 25 '10 at 22:43

Perhaps you could do a preliminary analysis on a random subset of the data to form hypotheses, and then test those few hypotheses of interest using the rest of the data. That way you would not have to correct for nearly as many multiple tests. (I think...)

Of course, if you use such a procedure you will be reducing the size of the dataset used for the final analysis and so reduce your power to find real effects. However, corrections for multiple comparisons reduce power as well and so I'm not sure that you would necessarily lose anything.

share|improve this answer
(+1) This is a great idea generally. For large correlation matrices, however, there are so many statistics and so many of them can simultaneously be spuriously large that it usually pays to adjust. Otherwise you wind up chasing a large number of misleadingly "significant" correlations that just disappear in the hold-out data. (Run a simulation with, say, a few hundred draws from 50 uncorrelated standard normal variates. It's an eye-opener.) –  whuber Dec 31 '10 at 5:14

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.