0
$\begingroup$

Is it incorrect to look at multiple testing, as applying a regression model with multiple variables? For example, I want to check wether there is a relationship between y and X1, X2, and X3. Then regression model would be unstable if there is correlation between the variables. Is that the same problem with multiple testing? That there is mixed effect while we check individually for every variable?

Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ If you have $k = 5$ levels of the factor in a one-way ANOVA, you are correct that there are ${5 \choose 2} = 10$ possible comparisons among levels--provided the main F-test finds not all means equal. The 10th post hoc test is no more likely to be significant that the first, but if you do ten tests, each at the 5% level, you have more than a 5% chance of finding a difference somewhere among 10 tests. $\endgroup$
    – BruceET
    Commented Nov 22, 2019 at 7:18
  • $\begingroup$ If you generate random noise data and do 10 hypothesis tests at p = 0.05, there will be far more than a 5% chance that at least one of the results will be siginficant. $\endgroup$
    – Peter Flom
    Commented Nov 22, 2019 at 23:19
  • $\begingroup$ The issue of there being correlation among your multiple independent variables (X1, X2, X3) is a separate issue from that of multiple testing. With either concern, it doesn't make either multiple regression or multiple simple regressions incorrect. It just requires some caution in interpretation. $\endgroup$
    – Sal Mangiafico
    Commented Nov 23, 2019 at 23:16

1 Answer 1

Reset to default
0
$\begingroup$

No, these are different issues.

The problem of multiple comparisons arises any time you do multiple comparisons. It could apply to correlations, t-tests, regressions or whatever. It has nothing to do with stability of coefficients.

In a regression with more than one independent variable, the regression estimates can be unstable if there is colinearity, which is not the same as correlation. That happens even if you do only one regression.

Each of these problems has been discussed here a lot.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.