Are t tests of coefficients in multiple regression post hoc tests?

Question

In multiple regression, if a global F test is significant, then are t tests (or Wald tests) for the coefficients considered to be multiple comparisons and post hoc tests and should they be adjusted?

Answer 1

General discussion of post hoc analyses

• post hoc analyses are typically contrasted with a priori, where post hoc analyses are, in some sense, performed after seeing the data, and a priori analyses are set out before seeing the data. In this sense the terms map on closely to the concepts of exploratory versus confirmatory data analysis.
• Arguably, the prototypical post hoc analysis involves comparing all pairwise comparisons of means in the context of an ANOVA where there are three or more levels to one of the independent variables. SPSS, for example, has a post hoc button specifically designed for running this form of analysis.
• However, while pairwise comparisons of group means is the prototypical example, any analysis that involves running statistical tests after seeing the data could be described as post hoc.
• When thinking about the general label "post hoc", it is helpful to think about its purpose. It is used as a caution about using standard inferential procedures on the problem at hand either because many significance tests are being run, or even if only a few significance tests are being performed, there are many that could have been performed had the data been different. Thus, post hoc statistical tests typically try to make adjustments to how inferences are made in order to control Type I error rates over both the analyses that are performed and those that could have been performed had the data been different.

The term "post hoc" applied to regression coefficients

• Thus, the examination of individual significance tests for regression coefficients could be labelled post hoc if the examination is done after seeing the data. Also, if you wanted to control your family wise error rate and you saw the set of significance tests associated with the regression coefficients as a family, then you could apply something like a Bonferroni adjustment to the individual significance tests. That said, many researchers might interpret an overall model more holistically, and it is ultimately up to the reader how they choose to interpret the significance tests associated with individual coefficients.
• I would not describe the examination of a set of significance tests for regression coefficients as "multiple comparisons", because I think the term "comparisons" pertains more to comparing group means. I'd prefer a term like "multiple significance tests".

Answer 2

It depends on your state of knowledge before the study. If you went into the study knowing that there were variables that were highly likely to be "significant" predictors of the outcome, and you were mostly interested in the influence of some new measure, say "M1", then the F-test is basically uninteresting and you are primarily interested in the relationship of M1 to the outcome. Then the relationship's features and statistical measures of credibility are not "post hoc", ... they are the primary study question.