# Correcting p-value in multiple regression [duplicate]

When running a multiple regression analysis, why do we not need to correct the p-values for the amount of predictors in the model?

summary(lm(mpg ~ disp + hp + drat + wt + gear, data=mtcars))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 27.733774   6.596345   4.204 0.000274 ***
disp         0.007368   0.011805   0.624 0.537998
hp          -0.041216   0.014317  -2.879 0.007882 **
drat         1.244929   1.490168   0.835 0.411088
wt          -3.406208   1.090465  -3.124 0.004351 **
gear         0.863456   1.110779   0.777 0.443972


Since each predictor is tested for significance individually, why is it ok to report these results without correcting for the amount of tests?

• Many thoughtful analysts do correct for the multiple testing.
– whuber
Mar 22 at 21:29
• Interesting. In my field nobody does, and I never understood why. I don't think I have ever seen a discussion on this, either in textbooks or here in CV. Are there any valid arguments why it might be ok not to correct? It just seems so pervasive... Mar 22 at 21:38

I suppose it depends on your intent.

In my day to day, I will regress an outcome on several variables but am really only interested in the effect of one. So I don't need to adjust anything -- those other variables are mainly there to reduce variance or so that I may stratify.

If you're conducting exploratory analyses and looking to see if anything is significant then yea it might be a good idea to correct the p value. However, my intuition says the degree to which you correct the p value would depend on the population correlation between covariates. Typical correction factors assume the tests are independent, but if the covariates are correlated I don't think this would be the case.

You can see this a little empirically. The following R code will simulate 1000 regressions and determine if any of the p values are less than 0.05

library(tidyverse)
library(broom)
library(rethinking)

sig <- rlkjcorr(1, 10, 1)
# sig <- diag(10)

replicate(1000, {

X <- MASS::mvrnorm(100, m = rep(0, 10), sig)
y <- rnorm(100)
lm(y~X) %>%
tidy %>%
pull(p.value)->p

any(p<0.05)

}) %>%
mean


I've included 2 lines for sig, which is intended to be the population covariance matrix for the covariates X. The probability that ANY of the p values would be less than 0.05 when variables are all independent (i.e. when sigma is the identity) is about 41%, and this decreases when we pick a random correlation matrix using rlkjcorr.

All in all, this means that typical p value correction methods (a la Bonferroni) might be too conservative and result in smaller type 1 error rate than might be desired.

I would argue that this depends on why you include multiple regressors in your model. Broadly speaking, two ends of a spectrum come to mind:

A) You are interested in the effect of one regressor and "only" include the others to hopefully avoid omitted variable bias (say, the effect of education on earnings, requiring you to control for things like ability and experience). Then, you would not really care about the effects of these regressors and their significances. Thus, despite having many regressors, you effectively still just conduct a single hypothesis test.

B) You are on a "fishing expedition" where you throw in a bunch of predictors and see which, if any, are related to the dependent variable. In my field (econometrics), "growth regressions" are a classical example, i.e., to find variables which predict why some countries grow fast and others do not. Predictors include all sorts of things, such as initial GDP, schooling levels, religion, geography,... You would then want to take multiplicity into account so as to avoid to spuriously find "relevant" variables just because you tried so many. [Something I have in fact done in a joint paper with my colleague Thomas Deckers.]

A reason that people might not be applying corrections is

• Because significance cut-off values are arbitrary anyway.

In some field there might be typically some couple of parameters being tested and if that number is not greatly varying then researchers in that field will settle on some value like the typical values as 0.01 or 0.05.

Maybe some field where researchers are testing more hypotheses at once in a single research (or more often in multiple researches) they will be using 0.01 more often instead of 0.05 and are indirectly using a correction in that way. There are even fields that use significance levels of 0.0000006 (the 5-sigma), and have no formal way of controlling for multiple regression (if you have multiple research groups doing the same experiment multiple times, how do you correct for that?).

• Because p-values are just a way to express error and people reading the results can imagine how this adds up for multiple tests done at once.

As an expression of error, p-values relate to the standard error and are a way to express that error in terms of a probability. Do we correct standard errors as well when we do multiple regression?

This is why researchers publish the actual p-values and not just whether something was below the (arbitrary) threshold or not, and any readers can do the maths themselves. (Often you have these star symbols added in tables with a legend like *** p<0.001 ** p<0.01 * p<0.05. That is just an aid for the readers, but the actual meaning of the cutoff values is a rule of thumb and not a strict rule)

If some research tested 20 variables and got one result with a p-value below 0.05 then readers will know that this is not a very reliable result.

• But you can make these two arguments in any scenario where multiple hypothesis correction may be applied. I agree with the OP's assessment that multiple hypothesis correction is performed in many circumstances, but not typically in multiple regression. What's different about this use case than any other application of multiple hypothesis correction? Mar 23 at 13:59
• @NuclearHoagie what's different is that with multiple regression one might do an omnibus test for all parameters as a whole, before analysing parameters seperately. Although I don't think that this is the reason why in practice people don't make multiple hypothesis corrections. I guess that it is just a custom/habit in particular fields. P-values are not strict decision rules anyway. You can compute it with rigorous mathematics, but underlying it are soft arbitrary assumptions and conventions. Mar 23 at 14:54

Yea, usually you need to because of the problem with multiple comparison (https://en.wikipedia.org/wiki/Multiple_comparisons_problem). Depending on what your are doing (academic or industry) the most conservative and considered the best is the Bonferroni correction (https://en.wikipedia.org/wiki/Bonferroni_correction). If you still have something after a Bonferroni correction, chances are it's robust. Here is an example of the analysis in R (https://rpubs.com/JLLJ/SPC12B).

• I wouldn't call Bonferroni 'the best' it is very conservative and reduces the power by a lot. Whoever uses this must be adjusting the significance level in order to make it work in practice. Mar 23 at 11:09
• *Whomever... And no, you don't have to adjust the significance level at all. It works just fine
– nick
Mar 24 at 12:52
• The adjusting of significance levels is done implicitly by basing the choice of cut-off values, which is an arbitrary choice, on the practical performance of experiments. In practice it's about a balance between type I and type II errors. When a more conservative correction is used, then there will be more often type II errors in comparison to type I errors, and it will just lead to a reconsideration of that balance. Mar 24 at 13:57