1
$\begingroup$

ANOVA is typically used to compare nested models. Is it necessary to control for multiple comparisons in this context? I'd say yes, because in practice we're performing multiple F-tests, and the more models we compare, the more likely it will be that we will find a significant difference among two models, even if there are none. Now, since the models are nested, I don't think we can consider the various hypotheses being tested to be independent. Thus I guess corrections such as Holm-Bonferroni, which make no assumption on the dependence of the F-values (or equivalently the p-values), should be used.

For non-nested models, we typically used AIC or BIC to compare models trained on the same data. Again, I think correcting for multiple testing is needed, if we are comparing many models, using methods which do not rely on special dependence structure among the hypotheses.

Is all this correct? In all the textbook examples I've seen where multiple models were compared, I've never seen the issue of multiple comparisons being considered, so I'm wondering if there's something I'm missing.

EDIT consider the following example:

foo <-data.frame(x=c(0.010355057, 0.013228936, 0.016313905, 0.019261687, 0.021710159, 0.023973474, 0.025968176, 0.027767232, 0.029459730, 0.030213807, 0.023582566, 0.008689883, 0.006558429, 0.005144958),
y=c(971.3800, 1025.2271, 1104.1505, 1034.2607, 902.6324, 713.9053, 621.4824, 521.7672, 428.9838, 381.4685, 741.7900, 979.7046, 1065.5245, 1118.0616))
Model1 <- lm(y~poly(x,1), data = foo)
Model2 <- lm(y~poly(x,2), data = foo)
Model3 <- lm(y~poly(x,3), data = foo)
Model4 <- lm(y~poly(x,4), data = foo)
Model5 <- lm(y~poly(x,5), data = foo)

anova(Model1, Model2, Model3, Model4, Model5)
> Analysis of Variance Table

Model 1: y ~ poly(x, 1)
Model 2: y ~ poly(x, 2)
Model 3: y ~ poly(x, 3)
Model 4: y ~ poly(x, 4)
Model 5: y ~ poly(x, 5)
  Res.Df    RSS Df Sum of Sq        F    Pr(>F)    
1     12 191485                                    
2     11  52318  1    139168 146.6072 2.002e-06 ***
3     10  44601  1      7717   8.1295 0.0214384 *  
4      9   7912  1     36689  38.6500 0.0002547 ***
5      8   7594  1       318   0.3349 0.5787128    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Thus, the differences between the degree 1 and degree 2 model, degree 2 and degree 3 model, degree 3 and degree 4 model are all significant, but the difference between the degree 4 and degree 5 model is not significant. I could stop here and choose Model4. However, I did perform 5 significance tests. Shouldn't I correct the alphas for the number of tests?

Note that here I made relatively few significance tests. But in Data Science/Machine Learning applications, you routinely compare thousands of models. Couldn't one model come out better than others, just by chance?

Improve this question
$\endgroup$
  • 1
    $\begingroup$ I don't understand why you think alpha error inflation occurs here. Can you explain that in more detail? $\endgroup$ – Roland Mar 1 '17 at 15:20
  • $\begingroup$ ok, I will add an example. $\endgroup$ – DeltaIV Mar 1 '17 at 15:34
  • $\begingroup$ @Roland done, please have a look if it's more clear now $\endgroup$ – DeltaIV Mar 1 '17 at 16:01
  • $\begingroup$ So, your actual problem is not model comparison but model selection. There are much better approaches for the latter. Instead of worrying about p-values look into techniques like the LASSO. See previous answers about model selection. $\endgroup$ – Roland Mar 1 '17 at 17:00
  • $\begingroup$ @Roland most of the questions I see where AIC or ANOVA are used, talk about model comparison: stats.stackexchange.com/questions/105003/…, stats.stackexchange.com/questions/100338/… [1/2] $\endgroup$ – DeltaIV Mar 1 '17 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.