As we know, if we are doing many tests or multiple comparison, we don't use the same $\alpha$ value and use some $\alpha$ correction methods like Bonferroni. This is done because when we do multiple tests, we have higher chance of getting something as significant compared to doing for fewer numbers of tests.

But my main question is this:

1) It is said if you are comparing multiple sample means using ANOVA and once you find there is some significant difference then you can do a post hoc analysis by doing pairwise comparison. But now you don't have to actually do a Bonferroni correction. Why is that? Isn't this post hoc analysis same as other pairwise t test where we use Bonferroni correction?

2) If Bonferroni correction is required because more tests leads to more chances of getting something significant then why we don't use the same thing, where we are doing something like regression where we are testing significance of $\beta$ estimates, or whether a variable is significant or not for feature selection using p value/F score? In that case also we are doing multiple comparison in checking whether each variable is significant or not. Then why don't we use Bonferroni correction on critical $\alpha$ there?

Please advise.


Preface: There are many different was to adjust for multiple comparisons. Olive Dunn proposed the Bonferroni adjustment in 1961, and the multiple comparisons literature (see, for example, Shaffer, 1995) has grown to a variety of family-wise error rate adjustment methods (of which Bonferroni is the simplest), and the more recent false discovery rate adjustment methods. Moreover, adjustments can either be made to $\alpha$, or the math may be inverted and instead applied to adjust p-values (sometimes adjusted p-values are called q-values)—my own preference is to adjust $\alpha$, since adjustments to p may need a clumsy upper-truncation at 1.0 to retain interpretability as a probability. Your question, and my answer applies regardless of which of these methods you choose, and whether you apply the adjustment to $\alpha$ or to the p-values.

  1. You would apply the Bonferroni to post hoc multiple comparisons following rejection of a one-way ANOVA. In fact that is a canonical example of when to apply the Bonferroni adjustment. These pairwise tests are not quite the same thing as a bunch of standard t tests, because following rejection of an ANOVA the t test statistics are calculated using the pooled variance implicit in the ANOVA's null hypothesis, rather than variance from the two specific groups compared for a single test statistic.

  2. You are correct: we would use multiple comparisons adjustments when make many statistical tests, as in the case of the t tests for the $\beta$ estimates in multiple regression, or in feature selection of N-way ANOVA.

Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64.

Shaffer, J. P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46:561–584.

  • $\begingroup$ But for the beta estimates in regression we don't do multiple comparison adjustments if I am right?. Like we see the same p value and alpha as 0.5 being reported by many softwares when we run regression analysis? My questions is why it is not corrected then? $\endgroup$ – Baktaawar May 20 '15 at 6:39
  • 1
    $\begingroup$ @user It depends on who you mean by "we". Certainly, there are lot's of folks who are willing to ignore the fact that the meaning of $\alpha$ changes when we conduct more than a single test... including more than a single test of $\beta$ coefficients. Many of these people are even published. I, and those whose statistical methods I respect do not ignore the fact of the meaning of $\alpha$ and what multiple tests do to it. I do indeed correct for multiple comparisons in a multiple regression context. Hence there are tools such as R's p.adjust(), and the qqvalue package for Stata, etc. $\endgroup$ – Alexis May 20 '15 at 17:42
  • 1
    $\begingroup$ @user As to why multiple comparisons are not automatically adjusted: there are many adjustment methods, and several considerations to make (e.g. what is a family in FWER methods, and comparisons across which sets of analysis?, etc.) that make "canning" of multiple comparisons adjustments less of an automated choice. $\endgroup$ – Alexis May 20 '15 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.