I am using the corr.test() function to do some exploratory data analysis. I am using the "holm" correction but I think I will switch to using BH. Eitherway I am curious about the p-value out put from this function. If the reported p-value is under 0.05 does that mean it is significant now?

  • $\begingroup$ Actually the BH method you mentioned might be one that controls for false positive rate instead of type I error. It will be more lenient. If you control false positive rate under 0.05, your family wise type I error can be much larger than 0.05 depending on the number of tests conducted. You need to make decision whether you want to control false positive rate or family wise type I error. $\endgroup$ – hehe Apr 4 at 16:55
  • $\begingroup$ I have 38 tests to run so setting my alpha to be around 1 in 40 at 0.025 should work correct? I'm not trying to find significant results, its more important that I do the test right in my opinion. $\endgroup$ – Angus Campbell Apr 4 at 17:33
  • $\begingroup$ The approach you use is bonferroni. It is certainly correct (will not inflate type I error) but is too conservative for so many tests. Holm will be better. $\endgroup$ – hehe Apr 4 at 17:49
  • $\begingroup$ Is there any way to objectively know what is best for me to use, or does it come down to a subjective call of I don't want to miss things (thus using something less conservative) vs I want to be strict and make sure I'm certain there is a correct correlation? $\endgroup$ – Angus Campbell Apr 4 at 18:11
  • $\begingroup$ Holm correction is strict in the sense that it ensures strong control of family wise type I error <=0.05. In your example, it is definitely a better choice than Bonferroni. more powerful. Bonferroni has the advantage of being separable, in the sense that if you fail some tests and win some, partial alpha can be passed down to test other things. But if you are not testing other endpoints, it does not matter. $\endgroup$ – hehe Apr 4 at 18:16

Holm and Hochberg are both used in multiple testing to adjust for p-values, so as to keep family wise type I error under 0.05. In the case of corr.test(), you usually test for many correlations at the same time, these adjusted p-value are really needed to make reliable conclusions/claim the correlation is statistically significant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.