Given that ANOVA is a special case of linear regression I recently wondered if we shouldn't use a correction for multiple testing (e.g. Bonferonni) as we would typically see in ANOVA.

As a very simple example suppose I am testing whether the wage for five job types (A, B,C,D,E) is the same on average, and if so how.

With an ANOVA procedure I would first run an ANOVA to test the null hypothesis: $$H_0:\mu^A=\mu^B=\mu^C=\mu^D=\mu^E$$ versus the alternative: $$H_1: H_0 \text{ is not true}$$

If I would then find a significant effect I would then continue through a range of pairwise t-tests, testing a.o. $\mu^A=\mu^B$, $\mu^C=\mu^D$, $\mu^A=\mu^C$ etc., to find which one is different from which, but typically I would have to correct for multiple testing through e.g. the Bonferroni correction.

In the linear regression set-up I would use wage as the dependent variable, and job-type dummies (or in my case I would use a factor in R) as the independent variables. For the sake of argument assume I use job type A as reference level.

The first thing to look at would be the F-test of the full model against the null model, which is of course the same as the ANOVA previously. Given that that is significant we would then look at the t-tests of the individual dummies, and see if their coefficients are significantly different from zero. This is in effect the post-hoc testing I would do in the ANOVA, although I do fewer tests because in this model I can only test: $\mu^A=\mu^B$, $\mu^A=\mu^C$, $\mu^A=\mu^D$ and $\mu^A=\mu^E$ given that A is the reference level.

My question then is: shouldn't we, strictly speaking, apply a correction for multiple testing here too?

I certainly haven't seen that done in any of the papers I've read.

  • $\begingroup$ It all depends on the purpose of your analysis. The Dunnet correction in post-hoc testing of treatment contrasts is e.g. not too exotic. $\endgroup$
    – Michael M
    Jun 15, 2017 at 20:22
  • $\begingroup$ @Michael M Well I was trying to ask a general question, because it surprised me that in my experience correction methods for multiple tests only get taught with ANOVA, but not with linear regression. Might just be my training. I didn't know Dunnet's correction but from the looks of it, it would be indeed more appropriate for linear regression than e.g. Bonferroni or Tukey $\endgroup$ Jun 15, 2017 at 21:11


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.