Does the computation of an N x N correlation matrix for N unrelated variables require multiple comparisons correction for all the computed pairwise correlations (assuming each computed correlation is a 'comparison' in the sense of being a 'statistical test')? If so, would FWER be a more appropriate measure than Bonferroni, or do any assumptions about the variables have to be made before using FWER? For instance, does it make a difference in what correction should be used whether the variables are a-priori likely to be correlated (e.g. test scores from the same group of subjects) vs uncorrelated (e.g. each variable represents the test score of a different sample)?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Please write out what "FWER" means so that people will not misunderstand (or simply skip over) your question. $\endgroup$– whuber ♦Commented Mar 14, 2014 at 15:29
-
3$\begingroup$ I assume you mean Family-Wise Error Rate by "FWER", but I don't follow the statement "would FWER be a more appropriate measure than Bonferroni", as Bonferroni is intended to control for family-wise error rate. $\endgroup$– gung - Reinstate MonicaCommented Mar 14, 2014 at 15:58
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
This depends on what question you are trying to answer and what your strategy is.
I like to think about what would happen to my conclusions if I were to add to my data some additional columns of randomly generated noise. In your case this would add more correlations.
If you will declare success/significance if any of the correlations are significant (fishing for significance) then yes, you need to do a correction for multiple comparisons because if the truth is nothing is correlated, but you add a bunch of random noise variables and don't adjust, then you will likely see something significant by chance.
If on the other hand there are specific comparisons that are of interest and would have been of interest if only those 2 variables had been in the study/dataset, then you probably don't want to adjust for multiple comparisons. Think about a case where 2 variables are correlated, but you would prefer them not to be (what I want to eat, but my wife doesn't want me to eat vs. a measure of my health), you could add a bunch of random noise variables and adjust for multiple comparisons and the adjustment would change a significant result into a non-significant result (great for justifying my snack, but not really honest).
answered Mar 14, 2014 at 15:50
-
$\begingroup$ Great answer, thank you Greg. So assuming you don't know much about the variables and just want to see who correlates with who, and so each cell in the corr matrix is of interest, which method would be most suitable: Bonferroni (e.g. divide by total number of pairwise correlations), or control of Family-wise Error Rate (FWER)? In fact, how would a 'family' be defined in the latter case? $\endgroup$– z8080Commented Mar 15, 2014 at 18:30
-
2$\begingroup$ The family wise error rate is just the probability of finding anything significant given that all the null hypotheses are true. In your case controlling the FWER just means that if none of the variables are correlated (all true cor's are 0) then the probability of finding one or more significant is 0.05 (or less) or whatever your FWER is. Bonferroni is one way to (conservatively) control the fWER. False Discovery Rate (FDR) is a little different and also tries to limit the number of false positives. $\endgroup$ Commented Mar 15, 2014 at 20:14