Should I adjust the significance levels in a multiple (linear) regression with dummy variables, when making many comparisons? I have an experiment with a continuous dependent variable, and an independent variable with a number of categories, k. The categories come from a randomised controlled trial I ran. There is 1 control group, and k-1 treatment groups.
I have created a multiple linear regression with k-1 dummy variables. My reference group is the control group. The control is the reference group because I want to test if there are significant differences between the control group, and all the treatment groups separately.
My question is the following: Do I need to adjust the significance level for any of the differences, to reduce the likelihood of a type I error?
In an ANOVA all comparisons are at a 95% percent level, and as far as I remember when there are more than one comparison, one must increase the significance level. Is it the same for a multiple regression with dummies?
Thank you
 A: Multiple degree of freedom composite ("chunk") tests have perfect multiplicities and are independent of the coding of the indicator variables (i.e., of choice of reference group).  Consistent with these composite tests (e.g., overall ANOVA), simultaneous confidence intervals are recommended.  Moving from individual confidence intervals of differences in means to simultaneous confidence intervals requires less of a "hit" than you might think, and covers all the bases if you are a frequentist.  Simultaneous confidence intervals are in a way more principled than ad hoc multiplicity adjustments because (1) they get us away from highly problematic dichotomous "significance" thinking and (2) they are unique, unlike the myriad multiplicity adjustments.  See for example the R rms package contrast.rms function which makes these intervals easy to compute for a variety of regression model types.
A: It depends on your approach, is it exploratory? If it is, then false positives are fine as long as you are open about this, so no adjustment required.
If it is confirmatory, then you should have directed hypotheses you want to test, and the tests should be sufficiently powered such that multiple correction if applied would not be a problem.
See Saville (1990), he explains why this approach is pretty reasonable. This passage captures his advice:

In the more general hypothesis testing context, the scenario that is most acceptable to statisticians is that of a well-designed study in which orthogonal contrasts are prespecified, corresponding to a "vision of reality" that will, it is hoped, be supported by the data. If this vision is not supported by the data, however, it is sometimes found that another set of orthogonal contrasts provides a good description of the data. This description generates a new vision of reality, which will then need to be confirmed in subsequent studies.
Multiple comparison procedures go against this basic philosophy in that they appear to formulate and test hypotheses in the same study simultaneously. In fact, the multiple comparison controversy is resolved if the procedures are thought of as hypothesis generators rather than as methods for simultaneous generation and testing.

Saville, D. J. (1990). Multiple Comparison Procedures: The Practical Solution. The American Statistician, 44(2), 174–180. Retrieved from http://www.jstor.org/stable/2684163
