I use Pearson's correlation to calculate association between Test A and performance on 3 subtests of test B. If I use Bonferroni correction for multiple comparison then alpha is 0.05/3 . we found significant r for one subtest of test B (p=0.02). But this will be rejected as per Bonferroni correction. Is there another way to do multiple comparison correction that is not as conservative for my data? These correlations are for the same group of people 1st correlation- Test A and Test B.1(r=-0.03, p=0.9, 95% CI= -0.45 to 0.4), 2nd correlation-Test A and Test B.2 (r=0.08, p=0.7, 95% CI= -0.35 to 0.49) 3rd correlation- Test A and Test B.3 (r=0.48, p=0.02, 95% CI= 0.08 to 0.74).
2 Answers
$\begingroup$
$\endgroup$
It seems that by "too conservative" what you mean is that it's not giving you the significance you want. Thinking of a method as too conservative "for your data" is incorrect. The conservativism of the method should be evaluated relative to the research question, not to the data. Waiting until the p-values have been computed and then choosing the multiple-testing method that gives you the answer you're looking for is not how good science is done. It might be better to be honest about the fact that you are in the early stages of researching the question and aren't prepared to draw conclusions from the data. You can follow up with a better-planned study.
$\begingroup$
$\endgroup$
A somewhat less conservative method for controlling family-wise error (probability of a false alarm/type 1 error) is to use a sequentially rejective method like Rom's Method. From the p values you've given, Rom's Method would declare the 3rd correlation significant, but the other 2 correlations would fail to reject. Chapter 13 of Rand Wilcox's Modern Statistics for the Social and Behavioral Science is a good source for these type of multiple comparisons procedures.
That said, trying out multiple-comparisons techniques one after another and going with the one that produces significant p-values is likely to produce false alarms/type 1 errors at a rate higher than the 5% that might be expected from your significance threshold. Here, the issue is less of a concern since there are only a few tests, but its still something to think about.
