Is there a multiple testing problem when performing t-tests for multiple coeffcients in linear regression?

Question

This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too?

Here's a common scenario that I've seen many times. A researcher runs a simple linear regression with, say, 5 covariates.

lm(Y ~ X1 + X2 + X3 + X4 + X5, data = df)

Ignoring interaction effects for the moment. They run the summary on the regression and observe the following:

      Estimate      Pr(>|t|)
X1    a            0.10
X2    b            0.47
X3    c            0.04            
X4    d            0.38
X5    e            0.12

From this, they conclude that covariate X3 is a significant predictor of outcome Y. I've seen this done many times.

My question is, why do we not have to adjust these $P$-values for multiple comparisons? Are we not doing 5 tests simultaneously, even though they are covariates, thus increasing the chances of seeing a false positive? Assuming 5 completely independent tests, there would be a $1-(1-\alpha)^M = 1-0.95^5 \sim 0.23$ or 23% chance of seeing a false positive, rather than than usual 5%, however this is in no way indicated in the reporting of the "significant association".

This article from the question Is adjusting p-values in a multiple regression for multiple comparisons a good idea? seems to indicate that if you are doing some kind of stepwise model selection, then it is advantageous to correct the $P$-values of your covariates to account for the increased type-1 error rate. This seems to indicate that tests covariates do not act differently than usual tests.

Has anyone had any experience with this? I would love to hear any flaws in my logic, or reasons why this should not be done.

Answer 1

For the multiple testing problem it might be good to take a look at Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?.

In your example above, if you estimate a regression on one sample, then you can, with a t-test only decide on the significance of an individual coefficient, so, yes, there is a multiple testing problem if you draw conclusions for multiple coefficients, based on multiple t-tests.

Let us call the coefficients $\beta_i, i = 1, 2, \dots 5$, then you can test $H_0^{(1)}: \beta_1 = 0$ versus $H_1^{(1)}: \beta_1 \ne 0$ with a t-test and conclude that $\beta_1$ is significant. Note that, if you can not reject $H_0^{(1)}$ that you can not conclude that $\beta_1$ is zero (see What follows if we fail to reject the null hypothesis?).

So if you want to find 'statistical evidence' for $\beta_1$ not being zero, then your $H_1^{(1)}$ must be the expression that you want to 'prove', i.e. $H_1^{(1)}: \beta_1 \ne 0$ and then $H_0^{(1)}$ is the opposite, i.e. $\beta_1=0$. As you assume $H_0^{(1)}$ to be true (to derive a statistical contradiction) you have a fixed value for the parameter $\beta_1=0$ and therefrom it follows that you know the distribution of the estimator $\hat{\beta}_1$ (see theory on linear regression) and you can compute p-values.

Let us now take the case where you want to show that $(\beta_1 \ne 0 \text{ and } \beta_2 \ne 0)$, then this must be your $H_1^{(1,2)}$ and the opposite $H_0^{(1,2)}$ is that either $(\beta_1 = 0 \text{ or } \beta_2 = 0)$, as there is an 'or' in there you can not fix all the parameters of the combined distribution of $(\hat{\beta}_1, \hat{\beta}_2)$ !

Can you apply multiple testing procedures ? Most of them assume that the individual p-values are independent, in this example $\hat{\beta}_1$ and $\hat{\beta}_2$ can not be shown to be independent !

But, in an advanced book on econometrics (e.g. W.H. Greene, "Econometric Analysis") you will find applicable test for J (simultaneous) linear restrictions ($\beta_i=0, i=1,2,3,4,5$ is a special type of 5 linear restrictions) that avoid the multiple testing problem.

Answer 2

There may be a few additional aspects worth considering (which are a little too long for a comment).

1. Whether or not there is a multiple testing problem in a given application quite strongly depends on which coefficients a researcher looks at. In many applications, one is only interested in 1-2 key variables, and the others only act as "controls". Say, in a fixed effects panel data model we may feel that we need individual specific intercepts to control for unobserved heterogeneity, but we are typically not really interested in these $N$ fixed effects per se. On the other hand, in for example growth econometrics, we sift through all possible determinants for growth and as such, we are willing to look at all significant variables. In the latter case, we do have a multiple testing problem, but not necessarily in the former.

2. I would argue that there are indeed several high-powered (at least, higher powered than Bonferroni) alternatives for performing such a model selection exercise. These include Bayesian model averaging, extreme-bounds analysis, General-to-specific, penalized methods (Lasso and related methods) and also methods directly deriving from the multiple testing literature. The latter group includes classical ones based on the Benjamini-Hochberg method, but also more recent bootstrap-based methods. To do some shameless self-promotion, these are compared and applied in a paper of mine.