Correcting p-value in multiple regression [duplicate]

Question

When running a multiple regression analysis, why do we not need to correct the p-values for the amount of predictors in the model?

summary(lm(mpg ~ disp + hp + drat + wt + gear, data=mtcars))

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 27.733774   6.596345   4.204 0.000274 ***
disp         0.007368   0.011805   0.624 0.537998
hp          -0.041216   0.014317  -2.879 0.007882 **
drat         1.244929   1.490168   0.835 0.411088
wt          -3.406208   1.090465  -3.124 0.004351 **
gear         0.863456   1.110779   0.777 0.443972

Since each predictor is tested for significance individually, why is it ok to report these results without correcting for the amount of tests?

Answer 1

I suppose it depends on your intent.

In my day to day, I will regress an outcome on several variables but am really only interested in the effect of one. So I don't need to adjust anything -- those other variables are mainly there to reduce variance or so that I may stratify.

If you're conducting exploratory analyses and looking to see if anything is significant then yea it might be a good idea to correct the p value. However, my intuition says the degree to which you correct the p value would depend on the population correlation between covariates. Typical correction factors assume the tests are independent, but if the covariates are correlated I don't think this would be the case.

You can see this a little empirically. The following R code will simulate 1000 regressions and determine if any of the p values are less than 0.05

library(tidyverse)
library(broom)
library(rethinking)

sig <- rlkjcorr(1, 10, 1)
# sig <- diag(10)

replicate(1000, {

  
X <- MASS::mvrnorm(100, m = rep(0, 10), sig)
y <- rnorm(100)
lm(y~X) %>% 
  tidy %>% 
  pull(p.value)->p


  any(p<0.05)

}) %>% 
  mean

I've included 2 lines for sig, which is intended to be the population covariance matrix for the covariates X. The probability that ANY of the p values would be less than 0.05 when variables are all independent (i.e. when sigma is the identity) is about 41%, and this decreases when we pick a random correlation matrix using rlkjcorr.

All in all, this means that typical p value correction methods (a la Bonferroni) might be too conservative and result in smaller type 1 error rate than might be desired.

Answer 2

I would argue that this depends on why you include multiple regressors in your model. Broadly speaking, two ends of a spectrum come to mind:

A) You are interested in the effect of one regressor and "only" include the others to hopefully avoid omitted variable bias (say, the effect of education on earnings, requiring you to control for things like ability and experience). Then, you would not really care about the effects of these regressors and their significances. Thus, despite having many regressors, you effectively still just conduct a single hypothesis test.

B) You are on a "fishing expedition" where you throw in a bunch of predictors and see which, if any, are related to the dependent variable. In my field (econometrics), "growth regressions" are a classical example, i.e., to find variables which predict why some countries grow fast and others do not. Predictors include all sorts of things, such as initial GDP, schooling levels, religion, geography,... You would then want to take multiplicity into account so as to avoid to spuriously find "relevant" variables just because you tried so many. [Something I have in fact done in a joint paper with my colleague Thomas Deckers.]

Answer 3

A reason that people might not be applying corrections is

• Because significance cut-off values are arbitrary anyway.

  In some field there might be typically some couple of parameters being tested and if that number is not greatly varying then researchers in that field will settle on some value like the typical values as 0.01 or 0.05.

  Maybe some field where researchers are testing more hypotheses at once in a single research (or more often in multiple researches) they will be using 0.01 more often instead of 0.05 and are indirectly using a correction in that way. There are even fields that use significance levels of 0.0000006 (the 5-sigma), and have no formal way of controlling for multiple regression (if you have multiple research groups doing the same experiment multiple times, how do you correct for that?).

• Because p-values are just a way to express error and people reading the results can imagine how this adds up for multiple tests done at once.

  As an expression of error, p-values relate to the standard error and are a way to express that error in terms of a probability. Do we correct standard errors as well when we do multiple regression?

  This is why researchers publish the actual p-values and not just whether something was below the (arbitrary) threshold or not, and any readers can do the maths themselves. (Often you have these star symbols added in tables with a legend like *** p<0.001 ** p<0.01 * p<0.05. That is just an aid for the readers, but the actual meaning of the cutoff values is a rule of thumb and not a strict rule)

  If some research tested 20 variables and got one result with a p-value below 0.05 then readers will know that this is not a very reliable result.

Answer 4

Yea, usually you need to because of the problem with multiple comparison (https://en.wikipedia.org/wiki/Multiple_comparisons_problem). Depending on what your are doing (academic or industry) the most conservative and considered the best is the Bonferroni correction (https://en.wikipedia.org/wiki/Bonferroni_correction). If you still have something after a Bonferroni correction, chances are it's robust. Here is an example of the analysis in R (https://rpubs.com/JLLJ/SPC12B).