After looking into Type 1 error correction protocols for multiple comparisons in ANOVA vs. MR, I found two (conflicting?) messages.
- The same Type 1 Error corrections that apply to comparisons in ANOVA apply to all comparisons in multiple regression.
- The Type 1 Error corrections that apply to multiple comparisons in ANOVA apply only non-orthogonal comparisons in multiple regression*.
Questions: What is the best practice, (even if it is not 1 or 2) for multiple comparisons in multiple regression? (Why?)
One textbook suggests that correcting for multiple comparisons in multiple regression is basically the same as correcting for multiple comparisons in ANOVA.
There is a large collection of statistical methods designed to cope with the problem of making all simple comparisons among g means. These vary in their definition of the problem, particularly in their conceptualization of Type I error, and they therefore vary in power and in their results. For example, the Tukey HSD test (Winer, 1971, pp. 197-198) controls the experimentwise error rate at ἀ. The Newman-Keuls test and the Duncan test […]. One of the oldest and simplest procedures for all pairs of g means is Fisher's "protected t" (or LSD) test (Carmer & Swanson, 1973). First, an ordinary (ANOVA) overall F test is performed on the set of g means (df = g — 1,n — g). If F is not significant, no pair-wise comparisons are made. Only if F is significant at the a criterion level are the means compared; this being done by an ordinary t test. The t tests are protected from large experiment-wise Type I error by the requirement that the preliminary F test must meet the a criterion. As we will see, this procedure is readily adapted for general use in MRC analysis. (Cohen & West, 2002, p. 183-184).
But then the textbook seems to suggest that multiple regression might somehow not need Type I error correction if its comparisons are orthogonal(ly coded):
3. Orthogonal comparisons. With g groups, it is possible to test up to g - 1 null hypotheses on comparisons (linear contrasts) that are orthogonal (i.e., independent of each other). These may be simple or complex. A complex comparison is one that involves more than two means, for example, M1 versus the mean of M3, M4, M5, or the mean of M1 and M2 versus the mean of M3 and M5. These two complex "mean of means" comparisons are, however, not orthogonal. (The criterion for orthogonality of contrasts and some examples are given in Chapter 8.) When the maximum possible number of orthogonal contrasts, g — 1, are each tested at ἀ, the experiment-wise Type I error rate is larger, specifically, it is approximately 1 - (1 - a) g-1 = .226. It is common practice, however, not to reduce the per-contrast rate ἀ below its customary value in order to reduce the experiment-wise rate when orthogonal contrasts are used (Games, 1971). Planned (a priori) orthogonal comparisons are generally considered the most elegant multiple comparison procedure and have good power characteristics, but alas, they can only infrequently be employed in behavioral science investigations because the questions to be put to the data are simply not usually independent (e.g., those described in paragraphs 1 and 2 previously discussed and in the next paragraph).
4. Nonorthogonal, many, and post hoc comparisons. Although only g - 1 orthogonal contrasts are mathematically possible, the total number of different mean of means contrasts is large, and the total number of different contrasts of all kinds is infinite for g > 2. An investigator may wish to make more than g — 1 (and therefore necessarily nonorthogonal) comparisons, or may wish to make comparisons that were not contemplated in advance of data collection, but rather suggested post hoc by the sample means found in the research. Such "data snooping" is an important part of the research process, but unless Type I error is controlled in accordance with this practice, the experiment-wise rate of spuriously "significant" t values on comparisons becomes unacceptably high. The Scheffe test (Edwards, 1972; Games, 1971; R. G. Miller, 1966) is designed for these circumstances. It permits all possible comparisons, orthogonal or nonorthogonal, planned or post hoc, to be made subject to a controlled experiment-wise Type I error rate. Because it is so permissive, however, in most applications it results in very conservative tests, i.e., in tests of relatively low power (Games, 1971). (Cohen & West, 2002, p. 184-185).
Even more recent methods papers seem to suggest that multiple regression comparisons with orthogonal codes somehow does not need correct for what ANOVA does.
Planned orthogonal contrasts are equivalent to independent questions asked to the data. Because of that independence, the current procedure is to act as if each contrast were the only contrast tested. This amounts to not using a correction for multiple tests. (Abdi & Williams, 2010, p. 6)
But even that more recent paper suggests that MR orthogonal codes is “equivalent” to its ANOVA counterpart. So ¯\_(ツ)_/¯
The classical approach corrects for multiple statistical tests (e.g., using a Sidak or Bonferroni correction), but essentially evaluates each contrast as if it were coming from a set of orthogonal contrasts. The multiple regression (or modern) approach evaluates each contrast as a predictor from a set of non-orthogonal predictors and estimates its specific contribution to the explanation of the dependent variable. The classical approach evaluates each contrast for itself, whereas the multiple regression approach evaluates each contrast as a member of a set of contrasts and estimates the specific contribution of each contrast in this set. For an orthogonal set of contrasts, the two approaches are equivalent.” (Abdi & Williams, 2010, p. 10).
Abdi, H., & Williams, L. (2010). Contrast analysis. Encyclopedia of Research Design, 1, 243–251. https://www.researchgate.net/profile/Lynne_Williams/publication/232659402_Contrast_analysis/links/5a1d5e0ba6fdcc0af326d0d8/Contrast-analysis.pdf
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2002). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, 3rd Edition (Third edition). Mahwah, N.J: Routledge. https://amzn.to/2UmiuMb