3
$\begingroup$

Correction for multiple testing (e.g. Bonferroni correction) is recommended when multiple statistical tests are performed on same data. In multiple regression analysis, multiple testing is integral part of procedure- to find which predictor variables are independently related to outcome variable.

Is some kind of correction applied in usual methods of analysis of Multiple regression? I have not read it being used (in my limited reading). If correction is not done in this situation, then why not?

I briefly searched internet but could find only this reference for possible correction in multiple regression testing: https://www.researchgate.net/post/Do_we_have_to_apply_multiple_testing_correction_for_multiple_linear_regression

Edit: Also what about other forms of multivariate analysis?

$\endgroup$
2
  • $\begingroup$ The last time I saw this come up on here, I remember someone saying that, in his field of psychology, the literature does not make such a correction but also laments the fact the no one does the correction, I think even in the same paper! $\endgroup$
    – Dave
    Commented May 26, 2020 at 11:43
  • $\begingroup$ Thanks users for making this a "Famous Question" with >10k views! $\endgroup$
    – rnso
    Commented May 30 at 13:11

1 Answer 1

14
$\begingroup$

(NB: I started to write this before the question was edited; I don't comment here on the "empirical aspect", i.e., what is done in the literature, but rather on what "should" be done.)

This is a good question, and I don't think there is a good answer.

What I think is the following. Generally in statistics things are less "black and white" and it is much less clearly determined what is "really right and wrong" to do in many situations, than the impression that is often given in statistics teaching.

Quite generally I don't think it's right to say "this-and-that has to be done in this-and-that situation". There are obvious mistakes that should be avoided, but what is "right" is almost always debatable.

Regarding correction for multiple testing: All these corrections have pros and cons. For example, the Bonferroni correction will make sure that the actual Type I error probability for the combined $H_0$ ("all involved $H_0$ are true") is smaller or equal than the nominal test level to the price that it is quite conservative, i.e., the power goes down and the probability for Type II errors goes up.

When running tests in multivariate regression, it is legitimate to use the Bonferroni or any other correction, which then has the well known implications, guaranteed Type I error probability, decreased power. Standard software packages give you the p-value, so it is easily possible to correct for multiple testing, and there is nothing wrong with it if you are happy to accept the disadvantages.

I'd agree that it is wrong to blindly interpret the test without correction as if there was no multiple testing issue. In fact you can occasionally find the remark in the literature that if tests are used for variable selection, p-values are invalid in the strict sense because multiple testing is going on and not taken into account. I'd interpret this rather differently, namely as an integrated method for variable selection that implicitly uses p-values as descriptive statistics rather than running formal tests that could be interpreted as such. One can then investigate how well this does as an overall method (which actually sometimes works well and sometimes not - unfortunately this is also not "black and white"), and if it works well, that's the justification; the p-values used on the fly should not be interpreted in the framework of standard error probabilities resulting from tests.

If we just look at the $p$ t-tests that are carried out routinely in multivariate regression with $p$ variables, without variable selection, these can be interpreted at nominal level without correction for multiple testing, but this implies that the Type I error probability only holds for every single one of them, not for all together. It also means that if the smallest p-value you get from 10 variables is 0.039, you should have on the radar that this isn't a convincing indication that anything is going on, which in this case however will also be shown to you in all likelihood by an insignificant F-test of the $H_0$ that all regression coefficients are zero.

So my conclusion is that you may or may not apply correction for multiple testing if you know what you are doing and avoid over-interpretation (which unfortunately is often not the case in the applied literature).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.