74
$\begingroup$

I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among these there should (statistically) be a high correlation, even if the data is completely random (each measure has only about 100 datapoints).

When I find a correlation, how do I include the information about how hard I looked for a correlation, into it?

I am not at a high level in statistics, so please bear with me.

Improve this question
$\endgroup$
3
  • 6
    $\begingroup$ This is a great example of why one needs multiple hypothesis testing. $\endgroup$
    – user88
    Commented Dec 26, 2010 at 13:44
  • 1
    $\begingroup$ Presumably one can use the permutation procedure to generate a null distribution for significance thresholds for the largest correlation, a different threshold for the second largest correlation, and so on. Hopefully this would only take a few hours in Python or R. (Ha! Famous last words.) But surely someone must already have done this and saved the code somewhere? $\endgroup$
    – user16977
    Commented Nov 19, 2012 at 0:40
  • 4
    $\begingroup$ @tmo R on this machine takes 18 seconds to obtain 1000 realizations of the null permutation distribution of the max correlation coefficient for a 300 by 100 matrix x: correl <- function(x, k=1) { n <- dim(x)[2] * (dim(x)[2]-1) / 2; v <- cor(x); sort(v[lower.tri(v)])[(n-k+1):n] }; sim <- replicate(1000, correl(apply(x,2,sample))) $\endgroup$
    – whuber ♦
    Commented Nov 19, 2012 at 17:45

4 Answers 4

Reset to default
78
$\begingroup$

This is an excellent question, worthy of someone who is a clear statistical thinker, because it recognizes a subtle but important aspect of multiple testing.

There are standard methods to adjust the p-values of multiple correlation coefficients (or, equivalently, to broaden their confidence intervals), such as the Bonferroni and Sidak methods (q.v.). However, these are far too conservative with large correlation matrices due to the inherent mathematical relationships that must hold among correlation coefficients in general. (For some examples of such relationships see the recent question and the ensuing thread.) One of the best approaches for dealing with this situation is to conduct a permutation (or resampling) test. It's easy to do this with correlations: in each iteration of the test, just randomly scramble the order of values of each of the fields (thereby destroying any inherent correlation) and recompute the full correlation matrix. Do this for several thousand iterations (or more), then summarize the distributions of the entries of the correlation matrix by, for instance, giving their 97.5 and 2.5 percentiles: these would serve as mutual symmetric two-sided 95% confidence intervals under the null hypothesis of no correlation. (The first time you do this with a large number of variables you will be astonished at how high some of the correlation coefficients can be even when there is no inherent correlation.)

When reporting the results, no matter what computations you do, you should include the following:

  • The size of the correlation matrix (i.e., how many variables you have looked at).

  • How you determined the p-values or "significance" of any of the correlation coefficients (e.g., left them as-is, applied a Bonferroni correction, did a permutation test, or whatever).

  • Whether you looked at alternative measures of correlation, such as Spearman rank correlation. If you did, also indicate why you chose the method you are actually reporting on and using.

Improve this answer
$\endgroup$
7
  • 3
    $\begingroup$ This is a pretty thorough descriptiion of p-value adjustment methods but what is left unsaid is the criteria for adjustment. Traditionally it has been familywise error rate. But that is a strict criterion and is not useful when you are looking at thousands of comparisons. In that case the false discovery rate first suggested by Benjamini is now commonly used. $\endgroup$
    – Michael R. Chernick
    Commented May 5, 2012 at 3:01
  • $\begingroup$ What if we just want to look at correlations of very well defined pairs of variables (e.g. $corr(x_1,y_1)$,...,$corr(x_n,y_n)$, where each $x_i$ and $y_i$ are variables) but don't care about all the other possible combinations (i.e., don't care about $corr(x_i,y_j)$ $\forall i \not= j$)? Do we still need a correction? $\endgroup$
    – Jase
    Commented Dec 16, 2012 at 16:19
  • $\begingroup$ @Jase Yes, you do. The amount of correction depends on the interrelationships among the variables. Simulation-based methods are about the only practicable way to determine these corrections. $\endgroup$
    – whuber ♦
    Commented Dec 16, 2012 at 16:31
  • $\begingroup$ Wow nice. Will this method that you discussed also correct the standard errors for serial correlation and heteroscedasticity issues? $\endgroup$
    – Jase
    Commented Dec 16, 2012 at 16:38
  • $\begingroup$ @Jase It would be difficult to interpret correlation coefficients in a heteroscedastic model. Your comment appears to refer to a linear model in a time series setting, rather than estimation of multivariate correlation coefficients. $\endgroup$
    – whuber ♦
    Commented Dec 16, 2012 at 17:23
13
$\begingroup$

From your follow up response to Peter Flom's question, it sounds like you might be better served by techniques that look at higher level structure in your correlation matrix.

Techniques like factor analysis, PCA, multidimensional scaling, and cluster analysis of variables can be used to group your variables into sets of relatively more related variables.

Also, you may want to think theoretically about what kind of structure should be present. When your number of variables is large and the number of observations is small, you are often better relying more on prior expectations.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ (+1) Good advice in light of the OP's comment to @Peter Flom's response. $\endgroup$
    – whuber ♦
    Commented Dec 31, 2010 at 5:16
9
$\begingroup$

This is an example of multiple comparisons. There's a large literature on this.

If you have, say, 100 variables, then you will have 100*99/2 =4950 correlations.

If the data are just noise, then you would expect 1 in 20 of these to be significant at p = .05. That's 247.5

Before going farther, though, it would be good if you could say WHY you are doing this. What are these variables, why are you correlating them, what is your substantive idea?

Or, are you just fishing for high correlations?

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ The reason why I wanted to do it like this was to have an open mind towards understanding my data, so maybe in a way I am fishing for correlations, which I did not think of before, for the purpose of getting enlightened. I am certainly not doing this to satisfy my boss or something abitrary. I would rather not get into the specifics of the data, as I want a general answer to this question, so I can use it in all situations in the future. $\endgroup$
    – David
    Commented Dec 25, 2010 at 22:43
9
$\begingroup$

Perhaps you could do a preliminary analysis on a random subset of the data to form hypotheses, and then test those few hypotheses of interest using the rest of the data. That way you would not have to correct for nearly as many multiple tests. (I think...)

Of course, if you use such a procedure you will be reducing the size of the dataset used for the final analysis and so reduce your power to find real effects. However, corrections for multiple comparisons reduce power as well and so I'm not sure that you would necessarily lose anything.

Improve this answer
$\endgroup$
1
  • 7
    $\begingroup$ (+1) This is a great idea generally. For large correlation matrices, however, there are so many statistics and so many of them can simultaneously be spuriously large that it usually pays to adjust. Otherwise you wind up chasing a large number of misleadingly "significant" correlations that just disappear in the hold-out data. (Run a simulation with, say, a few hundred draws from 50 uncorrelated standard normal variates. It's an eye-opener.) $\endgroup$
    – whuber ♦
    Commented Dec 31, 2010 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.