I fitted thousand of linear regression models and I corrected the p-values of the beta coefficients for multiple testing. When it's time to test goodness of fit of these models (normality, heteroskedasticity, lack of fit), does multiplicity come to play? What I am doing is Kolmogorv-Smirnov to test normality of the residuals of each model, does it make sense to adjust p-values for multiple testing to maintain 5% FWER? In this case a type II error is worse than a Type I error, specially in testing heteroskedasticity or lack-of-fit test. Thanks!
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
No, because p values aren't really the point of goodness of fit testing. The reason you test goodness of fit (e.g. here, that the residuals are normal) is that, if they are not, it violates an assumption of the model.
So, the key point is not whether the non-normality is significant, but whether it is large enough to invalidate any conclusions you might make from the model.
Much better to look at quantile normal plots.
answered Mar 9, 2017 at 12:12
-
$\begingroup$ Thank you very much, but in case of 1000 models, looking at QQplots is not very feasible... $\endgroup$– RobizMar 9, 2017 at 12:34
-
$\begingroup$ Actually, it's very feasible, it just takes time. If you look at the average plot for 30 seconds, then it would take 500 minutes or about 10 hours. But that's probably an over-estimate. However, there are other things you have to do for each model besides this. So, if you are running thousands of models, I'd say you should plan on it taking dozens of hours of your time. $\endgroup$ Mar 9, 2017 at 13:02