This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too?
Here's a common scenario that I've seen many times. A researcher runs a simple linear regression with, say, 5 covariates.
lm(Y ~ X1 + X2 + X3 + X4 + X5, data = df)
Ignoring interaction effects for the moment. They run the summary on the regression and observe the following:
Estimate Pr(>|t|) X1 a 0.10 X2 b 0.47 X3 c 0.04 X4 d 0.38 X5 e 0.12
From this, they conclude that covariate
X3 is a significant predictor of outcome
Y. I've seen this done many times.
My question is, why do we not have to adjust these $P$-values for multiple comparisons? Are we not doing 5 tests simultaneously, even though they are covariates, thus increasing the chances of seeing a false positive? Assuming 5 completely independent tests, there would be a $1-(1-\alpha)^M = 1-0.95^5 \sim 0.23$ or 23% chance of seeing a false positive, rather than than usual 5%, however this is in no way indicated in the reporting of the "significant association".
This article from the question Is adjusting p-values in a multiple regression for multiple comparisons a good idea? seems to indicate that if you are doing some kind of stepwise model selection, then it is advantageous to correct the $P$-values of your covariates to account for the increased type-1 error rate. This seems to indicate that tests covariates do not act differently than usual tests.
Has anyone had any experience with this? I would love to hear any flaws in my logic, or reasons why this should not be done.