# Should I adjust the significance levels in a multiple (linear) regression with dummy variables, when making many comparisons?

I have an experiment with a continuous dependent variable, and an independent variable with a number of categories, k. The categories come from a randomised controlled trial I ran. There is 1 control group, and k-1 treatment groups.

I have created a multiple linear regression with k-1 dummy variables. My reference group is the control group. The control is the reference group because I want to test if there are significant differences between the control group, and all the treatment groups separately.

My question is the following: Do I need to adjust the significance level for any of the differences, to reduce the likelihood of a type I error?

In an ANOVA all comparisons are at a 95% percent level, and as far as I remember when there are more than one comparison, one must increase the significance level. Is it the same for a multiple regression with dummies?

Thank you

Multiple degree of freedom composite ("chunk") tests have perfect multiplicities and are independent of the coding of the indicator variables (i.e., of choice of reference group). Consistent with these composite tests (e.g., overall ANOVA), simultaneous confidence intervals are recommended. Moving from individual confidence intervals of differences in means to simultaneous confidence intervals requires less of a "hit" than you might think, and covers all the bases if you are a frequentist. Simultaneous confidence intervals are in a way more principled than ad hoc multiplicity adjustments because (1) they get us away from highly problematic dichotomous "significance" thinking and (2) they are unique, unlike the myriad multiplicity adjustments. See for example the R rms package contrast.rms function which makes these intervals easy to compute for a variety of regression model types.

ANOVA and linear regression with only one independent variable that is categorical are equivalent. In this case F test from ANOVA is exactly the same as F test for whole regression.

So, you can run ANOVA and apply some post-hoc test. I'll recommend Dunnett's test because it compares one level of your independent variable to every other levels.

Toy example (with R):

First, I create random Y (my dependent variable) and X (independent) with 5 levels A, B, C, D and E:

> set.seed(123)
> y<-rnorm(100)
> x<-gl(5,20, labels = LETTERS[1:5])


Then I run ANOVA:

> anova(lm(y~x))
Analysis of Variance Table

Response: y
Df Sum Sq Mean Sq F value Pr(>F)
x          4  2.965 0.74117  0.8854 0.4758
Residuals 95 79.525 0.83711


And Dunnett test to compare A with B, C, D and E.

> library(multcomp)
> summary(glht(lm(y~x), linfct = mcp(x = "Dunnett")))

Simultaneous Tests for General Linear Hypotheses

Multiple Comparisons of Means: Dunnett Contrasts

Fit: lm(formula = y ~ x)

Linear Hypotheses:
Estimate Std. Error t value Pr(>|t|)
B - A == 0 -0.19288    0.28933  -0.667    0.907
C - A == 0 -0.03514    0.28933  -0.121    1.000
D - A == 0 -0.26154    0.28933  -0.904    0.776
E - A == 0  0.23347    0.28933   0.807    0.836
(Adjusted p values reported -- single-step method)


And, for comparison, linear regression with dummies:

summary(lm(y~x))

Call:
lm(formula = y ~ x)

Residuals:
Min       1Q   Median       3Q      Max
-2.18925 -0.59938 -0.03713  0.60848  2.17000

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.14162    0.20459   0.692    0.490
xB          -0.19288    0.28933  -0.667    0.507
xC          -0.03514    0.28933  -0.121    0.904
xD          -0.26154    0.28933  -0.904    0.368
xE           0.23347    0.28933   0.807    0.422

Residual standard error: 0.9149 on 95 degrees of freedom
Multiple R-squared:  0.03594,   Adjusted R-squared:  -0.004652
F-statistic: 0.8854 on 4 and 95 DF,  p-value: 0.4758


See that:

• F test from ANOVA and regression are the same
• test statistics for dummies are the same as Dunnett's statistics, but...
• p-values are different, since Dunnett's test corrects for multiple comparisons.
• Nice answer. However someone may interpret it as indicating that Duncan's should only be done if ANOVA is significant (I know you didn't say that). Dunnett's controls the Type I error rate and an ANOVA could have low power if the treatments are very similar. – David Lane Jul 26 '17 at 19:53
• Okay thank you for the answer. I will consider using the ANOVA with a Dunnett's test. My primary goal with the experiment is to compare my treatments performance to the control group. However, it would also be interesting to perform other comparisons between treatments. In your example, that could be C versus D. How can I correct for multiple comparisons manually? Just to clarify conceptually: If I were to use the multiple regression analysis instead of the ANOVA with a dunnett's test, would I also be able to correct the p-values for multiple comparisons manually? – john Jul 27 '17 at 14:03
• Dunett's do not perform everything-to-everything comparisons, just one-to-everything_else. So you just compare control to others with it. I don't know how to correct regression manually. – Łukasz Deryło Jul 27 '17 at 14:56
• Okay. I want to use both the multiple regression analysis and the ANOVA to test the hypothesis. If I run the ANOVA, and I correct for multiple comparisons, I only get one significant result. If I run the multiple regression analysis, and leave the significance levels as they are, I get 3 significant results. You mentioned: "ANOVA and linear regression with only one independent variable that is categorical are equivalent. ". If this is the case, then I have to correct errors in the multiple regression case, in order to reduce the type I error right? – john Jul 28 '17 at 9:34

It depends on your approach, is it exploratory? If it is, then false positives are fine as long as you are open about this, so no adjustment required.

If it is confirmatory, then you should have directed hypotheses you want to test, and the tests should be sufficiently powered such that multiple correction if applied would not be a problem.

See Saville (1990), he explains why this approach is pretty reasonable. This passage captures his advice:

In the more general hypothesis testing context, the scenario that is most acceptable to statisticians is that of a well-designed study in which orthogonal contrasts are prespecified, corresponding to a "vision of reality" that will, it is hoped, be supported by the data. If this vision is not supported by the data, however, it is sometimes found that another set of orthogonal contrasts provides a good description of the data. This description generates a new vision of reality, which will then need to be confirmed in subsequent studies.

Multiple comparison procedures go against this basic philosophy in that they appear to formulate and test hypotheses in the same study simultaneously. In fact, the multiple comparison controversy is resolved if the procedures are thought of as hypothesis generators rather than as methods for simultaneous generation and testing.

Saville, D. J. (1990). Multiple Comparison Procedures: The Practical Solution. The American Statistician, 44(2), 174–180. Retrieved from http://www.jstor.org/stable/2684163

• Thank you for the answer. I am performing an experiment from a published paper, with another task and in another setting. I expect to get different results, but because much of the implementation in my experiment is based on the published paper, I would guess the approach is confirmatory. – john Jul 27 '17 at 14:07
• You can choose to lower the alpha level to .01 voluntarily rather than adjusting using any multiple comparison. Since your approach is supposed to be confirmatory, it should be sufficiently powered such that you should be able to lower the alpha level. And since this is confirmatory, you should have directed hypotheses you want to test, not testing everything. – Heteroskedastic Jim Jul 27 '17 at 14:21
• Your specific hypotheses can be tested using planned (orthogonal) contrasts since your approach is confirmatory. This would be the gold standard. Most software allow you to set up and test planned contrasts. – Heteroskedastic Jim Jul 27 '17 at 14:31
• With the planned contrasts, I assume you are talking about an ANOVA? Do you know how to correct the significance level in a multiple regression with dummy variables, and if it is necessary at all? – john Jul 28 '17 at 8:58
• There's nothing that situates planned contrasts within ANOVA beyond convention. Software like R allows you to test planned contrasts in multiple regression. By "correct the significance level", you mean what? I've spent my whole answer explaining how there is no need for such adjustments. You could simply decide to use an alpha less than .05. I'd advice that you read the paper. – Heteroskedastic Jim Jul 28 '17 at 14:53