I have 50 variables in my dataset. I have correlated each variable against each variable, thus I have $49·50/2 = 1225$ unique correlations when no variable is tested against itself. Now, suppose I want to correct the statistics for multiple comparison, and that for some reason (well, just for learning now) I want to use the Bonferroni correction. Let the threshold for significance be $\alpha = 0.05$. Is the corrected threshold $0.05/1200$ (where 1225 is number of tests) or $0.05/49$ (because each variable was correlated with 49 other)?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Note that all corrections for multiple testing are overly conservative in your case as your tests are not fully independent: If $X$ is strongly correlated to $Y$ and $Y$ is strongly correlated to $Z$ then it is very likely that $X$ is also strongly correlated to $Z$. $\endgroup$– WrzlprmftCommented Sep 30, 2014 at 14:12
-
2$\begingroup$ @Wrzlprmft: There are multiplicity corrections that are designed for known dependency (in particular due to the design, like here) or estimate the dependency in order to be less conservative. $\endgroup$– Horst GrünbuschCommented Sep 30, 2014 at 15:55
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
With Bonferroni correction the divisor is equal to the number of tests you carry out, dependent or independent
It helps to understand the purpose of the Bonferroni correction. You are testing correlations between variables. Let’s assume your null hypothesis for any two variables is that the correlation is 0. (Any null hypothesis will suffice.) Your significance threshold is $\alpha = 0.05$. In other words, there is a 5% chance that you will reject the null hypothesis erroneously. This is known as a type 1 error or a false positive.
Now, let’s say you did 100 tests, all at the $\alpha = 0.05$ level. You would expect $5%$ of these to give a false positive ( ie to fail by chance alone). If you do 1225 tests then you expect 5% = ~61 false positives. This is quite a lot! Bonferroni offers a level of protection against this scenario. You can think of it as familywise protection as it offers a family of tests a single level of protection against even one false positive. Instead of testing with an $\alpha = 0.05$ threshold, you perform each test at the $\alpha = 0.5/1225 = ~.0000408$ threshold. In this case Bonferroni reduces the probability of even one false positive amongst all tests to an $\alpha = 0.05$ threshold.
Bonferonni is very conservative and there exists several improvements. My favourite is False Discovery Rate
-
$\begingroup$ Why the downvote? If there's an issue with the answer it'd be helpful to point it out - I can't see one $\endgroup$– martinoCommented Oct 10, 2014 at 11:44
$\begingroup$
$\endgroup$
If you are correcting for Multiple testing, you need to take into account the number of tests performed.
As you stated you don't need to correct for all $50*50 = 2500$ tests since $corr(X,X)=1$ and $corr(X,Y)=corr(Y,X)$. You need to correct for $(n-1)*n/2 = (49*50)/2 = 1225$ (note not $1200$ as you specified in question) tests.
Note if you are in a regression setting, checking for correlatin between your explanatory variables is basically testing for multicollinearity. There are more standard methods of looking into this problem that you should read up on (e.g. Variance inflation factors, Farrar–Glauber test).
answered Sep 30, 2014 at 13:10
$\begingroup$
$\endgroup$
2
Firstly, use the modified Holm-Bonferroni method instead of the plain Bonferroni method.
Regarding the 49 vs 1200 question. In my opinion, if you want to make statistical assumptions on all the variables together, then you have to use the 1200 number. The number 49 would come in order if you wanted to test whether a single variable is somehow different from the others.
But still, both numbers are very high and I think you will have problems with the corrected α values as they will be ridiculously small.
Maybe a change of method is in order, as pairwise comparison is definitely not the best option here. Maybe some other method, which could test on all the variables altogether - like ANOVA or Friedman test (but I doubt that those two will be relevant for your case).
-
$\begingroup$ @HorstGrünbusch I am pretty sure it's not the case. Incidentally, if it is, all the implementations I know (e.g. R's
p.adjust) are wrong because you certainly don't need to specify any order. $\endgroup$– GalaCommented Sep 30, 2014 at 14:16 -
$\begingroup$ You're right, it's ordering on its own by the $p$-values. So you don't have to prespecify them. $\endgroup$ Commented Sep 30, 2014 at 14:55
$\begingroup$
$\endgroup$
1
if you are trying to find out whether a p-value corresponded to a correlation coefficient is significant or not using Bonferroni correction, it will be right to use the following correction: 0.05/((2500-50)/2) = 0.05/1225
-
1$\begingroup$ Although that is the right formula for correcting $1225$ nearly-independent tests, the huge degree of interdependence among the correlations of $50$ variables would suggest including a strong cautionary note against relying on the Bonferroni method in this situation (except perhaps in interpreting its result as a lower bound on the correct p-value). $\endgroup$– whuber ♦Commented Oct 1, 2014 at 14:58